2024-09-16T13:17:18.151 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T13:17:18.167 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T13:17:18.252 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writethrough} email: sangadi@redhat.com first_in_suite: false job_id: '7907305' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: rbd cache: true rbd cache max dirty: 0 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMO+gKCWgKmEx2hg9z3aH8/y+UKBc3gY4+qAQVhKx+mFWZuxbYwVhFTECXkmypzj55iAvj4fGTiS0chpy0sKWY8= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0IU3QQdFnLPQv8FQWY3up2buWqKw+C3q/al6rb/lW3u+UiB2EpvfU5ap4y+TscP5jI4NcqaxugdM4GYZ26Urg= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T13:17:18.252 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T13:17:18.253 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T13:17:18.253 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T13:17:18.255 INFO:teuthology.task.internal:Checking packages... 2024-09-16T13:17:18.276 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T13:17:18.276 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T13:17:18.276 INFO:teuthology.packaging:ref: None 2024-09-16T13:17:18.276 INFO:teuthology.packaging:tag: None 2024-09-16T13:17:18.276 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T13:17:18.276 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:17:18.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T13:17:18.422 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-16T13:17:18.423 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T13:17:18.431 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T13:17:18.431 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T13:17:18.439 INFO:teuthology.task.internal:Saving configuration 2024-09-16T13:17:18.455 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T13:17:18.463 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T13:17:18.487 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 13:11:45.684987', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMO+gKCWgKmEx2hg9z3aH8/y+UKBc3gY4+qAQVhKx+mFWZuxbYwVhFTECXkmypzj55iAvj4fGTiS0chpy0sKWY8='} 2024-09-16T13:17:18.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 13:11:45.683330', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0IU3QQdFnLPQv8FQWY3up2buWqKw+C3q/al6rb/lW3u+UiB2EpvfU5ap4y+TscP5jI4NcqaxugdM4GYZ26Urg='} 2024-09-16T13:17:18.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T13:17:18.516 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-16T13:17:18.516 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-16T13:17:18.516 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T13:17:18.579 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1fc708bac0>, signals=[15]) 2024-09-16T13:17:18.579 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T13:17:18.586 INFO:teuthology.task.internal:Opening connections... 2024-09-16T13:17:18.586 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:17:18.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:17:18.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:17:18.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:17:18.740 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T13:17:18.748 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-09-16T13:17:18.765 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-09-16T13:17:18.766 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-09-16T13:17:18.822 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-09-16T13:17:18.823 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T13:17:18.823 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-09-16T13:17:18.823 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T13:17:18.823 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T13:17:18.823 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T13:17:18.823 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-09-16T13:17:18.846 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-09-16T13:17:18.865 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-09-16T13:17:18.865 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T13:17:18.925 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-09-16T13:17:18.926 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T13:17:18.926 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T13:17:18.926 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T13:17:18.926 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-09-16T13:17:18.946 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T13:17:18.960 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T13:17:18.998 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T13:17:18.998 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-09-16T13:17:19.001 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-09-16T13:17:19.018 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T13:17:19.036 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T13:17:19.036 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-09-16T13:17:19.057 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-09-16T13:17:19.103 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T13:17:19.162 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T13:17:19.178 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T13:17:19.178 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T13:17:19.179 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T13:17:19.179 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T13:17:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:17:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:17:19.198 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-09-16T13:17:19.198 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T13:17:19.255 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-503.el9.x86_64 2024-09-16T13:17:19.256 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-503.el9.x86_64 2024-09-16T13:17:19.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-09-16T13:17:19.257 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-503.el9.x86_64 2024-09-16T13:17:19.257 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-503.el9.x86_64 2024-09-16T13:17:19.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-09-16T13:17:23.143 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-09-16T13:17:24.464 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.2 MB 00:03 2024-09-16T13:17:24.927 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-16T13:17:26.498 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T13:17:30.827 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.5 MB 00:02 2024-09-16T13:17:30.974 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-16T13:17:32.386 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-16T13:17:32.499 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-16T13:17:34.015 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-16T13:17:34.078 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-16T13:17:39.429 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T13:17:39.567 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T13:17:41.518 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:17:41.526 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:17:41.526 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T13:17:41.526 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:17:41.526 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-09-16T13:17:41.527 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:17:41.528 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 79 M 2024-09-16T13:17:41.528 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 126 M 2024-09-16T13:17:41.528 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:17:41.557 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T13:17:41.557 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T13:17:41.640 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:17:41.648 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 79 M 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 126 M 2024-09-16T13:17:41.650 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:17:41.705 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 286 kB/s | 23 kB 00:00 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 289 kB/s | 23 kB 00:00 2024-09-16T13:17:42.198 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-16T13:17:42.352 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-16T13:17:42.753 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 41 MB/s | 35 MB 00:00 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-16T13:17:43.103 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-16T13:17:43.103 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:17:43.103 INFO:teuthology.orchestra.run.smithi163.stdout:Total 54 MB/s | 79 MB 00:01 2024-09-16T13:17:43.290 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 18 MB/s | 29 MB 00:01 2024-09-16T13:17:43.290 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:17:43.290 INFO:teuthology.orchestra.run.smithi076.stdout:Total 45 MB/s | 79 MB 00:01 2024-09-16T13:17:43.788 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:17:43.975 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:17:44.128 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:17:44.128 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:17:44.353 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:17:44.353 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:17:45.091 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:17:45.091 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:17:45.315 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:17:45.316 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:17:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:17:47.797 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:17:48.314 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T13:17:48.404 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:17:48.532 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T13:17:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:17:51.212 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:17:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:17:51.605 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:17:56.181 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:17:56.355 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:17:56.464 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:01.302 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.659 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.699 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.811 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.864 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.984 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:39.984 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T13:18:39.984 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:18:39.984 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:18:40.262 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:40.263 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T13:18:40.263 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T13:18:40.263 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.118 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.119 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.119 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:18:41.119 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:18:41.162 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.249 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.249 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.249 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T13:18:41.249 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T13:18:41.249 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T13:18:41.250 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T13:18:41.250 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.309 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-09-16T13:18:41.346 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 82.el9 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8274670 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T13:18:41.347 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T13:18:41.348 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T13:18:41.349 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-09-16T13:18:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:18:41.349 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T13:18:41.350 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T13:18:41.350 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T13:18:41.493 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T13:18:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.579 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.579 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T13:18:41.579 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T13:18:41.579 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T13:18:41.579 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T13:18:41.579 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T13:18:41.638 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T13:18:41.638 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-09-16T13:18:41.671 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-09-16T13:18:41.671 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-09-16T13:18:41.671 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-09-16T13:18:41.671 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 82.el9 2024-09-16T13:18:41.671 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8274670 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-09-16T13:18:41.672 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:18:41.673 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T13:18:41.674 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T13:18:41.674 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-505.el9.x86_64 2024-09-16T13:18:41.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T13:18:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-09-16T13:18:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-09-16T13:18:43.324 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T13:18:43.349 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-09-16T13:18:43.351 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T13:18:43.367 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.GLq2JtqwBA 2024-09-16T13:18:43.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GLq2JtqwBA 2024-09-16T13:18:43.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.GLq2JtqwBA 2024-09-16T13:18:43.574 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.GLq2JtqwBA is 6KB 2024-09-16T13:18:43.623 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.GLq2JtqwBA 2024-09-16T13:18:43.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T13:18:43.639 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T13:18:43.667 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-09-16T13:18:43.669 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-09-16T13:18:43.684 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.fDc88PCerY 2024-09-16T13:18:43.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fDc88PCerY 2024-09-16T13:18:43.701 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T13:18:43.702 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T13:18:43.702 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T13:18:43.702 INFO:teuthology.orchestra.run.smithi076.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T13:18:43.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T13:18:43.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.fDc88PCerY 2024-09-16T13:18:43.890 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.fDc88PCerY is 6KB 2024-09-16T13:18:43.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-09-16T13:18:43.939 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.fDc88PCerY 2024-09-16T13:18:43.954 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T13:18:44.017 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T13:18:44.017 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T13:18:44.017 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T13:18:44.018 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T13:18:44.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T13:18:44.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-09-16T13:19:13.913 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:19:13.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:14.172 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T13:19:14.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:19:14.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:32.417 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T13:19:32.673 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T13:19:41.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:19:41.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:41.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:19:41.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:44.481 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T13:19:44.737 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T13:19:56.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:19:56.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:56.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:19:56.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:19:59.553 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T13:19:59.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T13:20:14.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:20:14.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:20:14.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:20:14.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:20:33.345 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T13:20:46.631 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T13:20:47.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:20:47.227 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T13:20:47.227 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T13:20:47.273 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-505.el9.x86_64 2024-09-16T13:20:47.273 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T13:20:47.273 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T13:20:47.273 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T13:20:48.275 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T13:20:48.275 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T13:20:48.344 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2024-09-16T13:20:48.443 DEBUG:teuthology.parallel:result is None 2024-09-16T13:20:51.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:20:51.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:20:51.782 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-09-16T13:20:52.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:20:52.315 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T13:20:52.315 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-09-16T13:20:52.357 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-505.el9.x86_64 2024-09-16T13:20:52.358 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T13:20:52.358 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T13:20:52.358 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T13:20:53.358 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T13:20:53.358 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T13:20:53.420 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-09-16T13:20:53.575 DEBUG:teuthology.parallel:result is None 2024-09-16T13:20:53.576 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T13:20:53.583 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T13:20:53.583 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T13:20:53.586 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T13:20:53.601 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T13:20:53.607 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T13:20:53.616 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T13:20:53.616 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T13:20:53.644 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T13:20:53.693 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T13:20:53.699 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T13:20:53.700 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T13:20:53.702 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T13:20:53.731 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:20:53.740 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:20:53.743 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:20:53.750 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:20:53.752 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T13:20:53.759 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T13:20:53.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T13:20:53.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T13:20:53.816 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T13:20:53.825 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T13:20:53.825 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T13:20:53.857 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T13:20:53.873 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:20:53.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:20:54.014 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:20:54.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:20:54.143 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:20:54.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T13:20:54.212 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:20:54.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:20:54.329 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:20:54.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:20:54.410 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:20:54.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T13:20:54.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-09-16T13:20:54.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-09-16T13:20:54.520 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:20:54.565 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:20:54.856 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T13:20:54.864 INFO:teuthology.task.internal:Starting timer... 2024-09-16T13:20:54.864 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T13:20:54.874 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T13:20:54.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-09-16T13:20:54.931 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-09-16T13:20:54.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-09-16T13:20:54.979 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-09-16T13:20:54.981 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T13:20:54.981 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-09-16T13:20:55.024 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-09-16T13:20:55.025 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-09-16T13:20:55.045 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-09-16T13:20:55.045 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-09-16T13:20:55.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:20:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:20:55.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:20:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:20:55.121 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T13:20:55.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-09-16T13:20:55.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-09-16T13:20:55.213 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T13:20:55.223 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T13:20:55.233 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T13:20:55.234 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T13:24:36.907 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-09-16T13:24:36.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:24:36.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:24:36.986 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T13:24:37.061 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T13:24:37.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:24:37.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:24:37.137 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-09-16T13:24:37.215 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T13:24:37.215 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T13:24:37.226 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T13:24:37.227 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T13:24:37.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:24:37.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T13:24:37.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:24:37.257 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T13:24:37.272 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T13:24:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T13:24:37.296 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-09-16T13:24:37.299 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T13:24:37.308 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-09-16T13:24:37.322 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T13:24:37.323 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-09-16T13:24:37.335 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-09-16T13:24:37.336 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T13:24:37.349 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T13:24:37.363 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T13:24:37.391 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:24:37.395 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:24:37.395 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-16T13:24:37.395 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.395 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.396 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.396 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.430 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.434 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:24:37.435 INFO:teuthology.run_tasks:Running task install... 2024-09-16T13:24:37.443 DEBUG:teuthology.task.install:project ceph 2024-09-16T13:24:37.443 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T13:24:37.444 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-16T13:24:37.444 INFO:teuthology.task.install:Using flavor: default 2024-09-16T13:24:37.449 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T13:24:37.449 INFO:teuthology.task.install:extra packages: [] 2024-09-16T13:24:37.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T13:24:37.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T13:24:38.006 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:44 ago on Mon 16 Sep 2024 01:23:53 PM UTC. 2024-09-16T13:24:38.033 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:44 ago on Mon 16 Sep 2024 01:23:53 PM UTC. 2024-09-16T13:24:38.091 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T13:24:38.120 INFO:teuthology.orchestra.run.smithi163.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T13:24:38.134 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:24:38.135 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:24:38.135 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:24:38.162 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:24:38.163 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:24:38.163 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:24:38.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo dnf -y copr enable ceph/el9 2024-09-16T13:24:38.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo dnf -y copr enable ceph/el9 2024-09-16T13:24:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T13:24:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:, 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T13:24:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:of the main distribution, and quality may vary. 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:, 2024-09-16T13:24:38.414 INFO:teuthology.orchestra.run.smithi163.stderr:and packages are not held to any quality or security level. 2024-09-16T13:24:38.415 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T13:24:38.415 INFO:teuthology.orchestra.run.smithi163.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T13:24:38.415 INFO:teuthology.orchestra.run.smithi163.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T13:24:38.453 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2024-09-16T13:24:38.472 INFO:teuthology.orchestra.run.smithi163.stdout:Repository successfully enabled. 2024-09-16T13:24:38.491 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T13:24:38.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:24:38.507 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T13:24:38.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:24:38.651 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T13:24:38.651 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T13:24:38.654 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T13:24:38.654 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T13:24:38.762 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T13:24:38.762 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:24:38.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T13:24:38.767 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T13:24:38.767 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:24:38.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T13:24:38.788 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T13:24:38.788 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T13:24:38.792 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T13:24:38.792 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T13:24:38.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T13:24:38.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T13:24:38.929 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T13:24:38.933 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T13:24:38.993 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-09-16T13:24:38.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-09-16T13:24:38.997 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-09-16T13:24:38.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-09-16T13:24:39.225 INFO:teuthology.orchestra.run.smithi076.stdout:45 files removed 2024-09-16T13:24:39.233 INFO:teuthology.orchestra.run.smithi163.stdout:45 files removed 2024-09-16T13:24:39.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw 2024-09-16T13:24:39.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-09-16T13:24:39.801 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 290 kB/s | 77 kB 00:00 2024-09-16T13:24:39.826 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 278 kB/s | 77 kB 00:00 2024-09-16T13:24:40.084 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-16T13:24:40.094 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-16T13:24:40.352 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-16T13:24:40.353 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 8.0 kB/s | 1.8 kB 00:00 2024-09-16T13:24:40.519 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-16T13:24:40.537 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for el9 owned by ceph 51 kB/s | 7.6 kB 00:00 2024-09-16T13:24:41.212 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-16T13:24:42.178 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-09-16T13:24:44.404 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-16T13:24:48.775 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T13:24:50.631 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 19 kB 00:00 2024-09-16T13:24:51.868 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T13:24:57.341 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T13:24:59.833 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:24:59.843 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T13:24:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T13:24:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout:Install 22 Packages 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 2 Packages 2024-09-16T13:24:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:24:59.847 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 58 M 2024-09-16T13:24:59.847 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:00.676 INFO:teuthology.orchestra.run.smithi076.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 7.7 MB/s | 5.0 MB 00:00 2024-09-16T13:25:00.751 INFO:teuthology.orchestra.run.smithi076.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 337 kB/s | 25 kB 00:00 2024-09-16T13:25:00.843 INFO:teuthology.orchestra.run.smithi076.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-16T13:25:00.926 INFO:teuthology.orchestra.run.smithi076.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 5.4 MB/s | 458 kB 00:00 2024-09-16T13:25:01.168 INFO:teuthology.orchestra.run.smithi076.stdout:(5/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 20 MB/s | 22 MB 00:01 2024-09-16T13:25:01.310 INFO:teuthology.orchestra.run.smithi076.stdout:(6/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 11 MB/s | 14 MB 00:01 2024-09-16T13:25:01.335 INFO:teuthology.orchestra.run.smithi076.stdout:(7/24): python3-ceph-argparse-17.2.7-1643.gd75f 270 kB/s | 45 kB 00:00 2024-09-16T13:25:01.360 INFO:teuthology.orchestra.run.smithi076.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 2.4 MB/s | 120 kB 00:00 2024-09-16T13:25:01.377 INFO:teuthology.orchestra.run.smithi076.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.7 MB/s | 156 kB 00:00 2024-09-16T13:25:01.411 INFO:teuthology.orchestra.run.smithi076.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.2 MB/s | 320 kB 00:00 2024-09-16T13:25:01.437 INFO:teuthology.orchestra.run.smithi076.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-16T13:25:01.461 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:19 2024-09-16T13:25:01.462 INFO:teuthology.orchestra.run.smithi076.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-16T13:25:01.537 INFO:teuthology.orchestra.run.smithi076.stdout:(13/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_ 8.1 MB/s | 4.9 MB 00:00 2024-09-16T13:25:01.638 INFO:teuthology.orchestra.run.smithi076.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-09-16T13:25:01.663 INFO:teuthology.orchestra.run.smithi076.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 710 kB/s | 160 kB 00:00 2024-09-16T13:25:01.705 INFO:teuthology.orchestra.run.smithi076.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T13:25:01.730 INFO:teuthology.orchestra.run.smithi076.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-16T13:25:01.780 INFO:teuthology.orchestra.run.smithi076.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-09-16T13:25:01.814 INFO:teuthology.orchestra.run.smithi076.stdout:(19/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-16T13:25:01.839 INFO:teuthology.orchestra.run.smithi076.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T13:25:01.864 INFO:teuthology.orchestra.run.smithi076.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 804 kB/s | 67 kB 00:00 2024-09-16T13:25:01.940 INFO:teuthology.orchestra.run.smithi076.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T13:25:02.040 INFO:teuthology.orchestra.run.smithi076.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 17 MB/s | 2.9 MB 00:00 2024-09-16T13:25:02.090 INFO:teuthology.orchestra.run.smithi076.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 13 MB/s | 3.1 MB 00:00 2024-09-16T13:25:02.095 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:02.095 INFO:teuthology.orchestra.run.smithi076.stdout:Total 26 MB/s | 58 MB 00:02 2024-09-16T13:25:02.158 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:02.191 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:02.191 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:04.078 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:04.448 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T13:25:04.526 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:04.618 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:04.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T13:25:04.727 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:04.816 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:04.867 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T13:25:04.907 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T13:25:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T13:25:05.371 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T13:25:05.442 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T13:25:05.479 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T13:25:05.523 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T13:25:05.560 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T13:25:05.621 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T13:25:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T13:25:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T13:25:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T13:25:05.774 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T13:25:05.808 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T13:25:06.101 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T13:25:06.145 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T13:25:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T13:25:06.309 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T13:25:06.339 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T13:25:06.912 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-16T13:25:08.404 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-16T13:25:08.785 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:09.355 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:09.389 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T13:25:09.483 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T13:25:09.483 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T13:25:09.484 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T13:25:09.484 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:09.540 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:09.858 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-16T13:25:15.489 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T13:25:17.940 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:25:17.954 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T13:25:17.955 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T13:25:17.956 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout:Install 22 Packages 2024-09-16T13:25:17.957 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-09-16T13:25:17.958 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:17.958 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 58 M 2024-09-16T13:25:17.958 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:25:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:19.413 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-09-16T13:25:19.413 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-09-16T13:25:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:20.074 INFO:teuthology.orchestra.run.smithi163.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 2.7 MB/s | 5.0 MB 00:01 2024-09-16T13:25:20.174 INFO:teuthology.orchestra.run.smithi163.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 251 kB/s | 25 kB 00:00 2024-09-16T13:25:20.675 INFO:teuthology.orchestra.run.smithi163.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 1.2 MB/s | 637 kB 00:00 2024-09-16T13:25:20.760 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:25:20.822 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:20.824 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T13:25:20.833 INFO:teuthology.orchestra.run.smithi163.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.8 MB/s | 458 kB 00:00 2024-09-16T13:25:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T13:25:20.913 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:22.067 INFO:teuthology.orchestra.run.smithi163.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 4.0 MB/s | 4.9 MB 00:01 2024-09-16T13:25:22.133 INFO:teuthology.orchestra.run.smithi163.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 677 kB/s | 45 kB 00:00 2024-09-16T13:25:22.171 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:22.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T13:25:22.172 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T13:25:22.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T13:25:22.209 INFO:teuthology.orchestra.run.smithi163.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 1.6 MB/s | 120 kB 00:00 2024-09-16T13:25:22.284 INFO:teuthology.orchestra.run.smithi163.stdout:(8/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.0 MB/s | 156 kB 00:00 2024-09-16T13:25:22.485 INFO:teuthology.orchestra.run.smithi163.stdout:(9/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 3.4 MB/s | 14 MB 00:04 2024-09-16T13:25:22.510 INFO:teuthology.orchestra.run.smithi163.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 1.4 MB/s | 320 kB 00:00 2024-09-16T13:25:22.552 INFO:teuthology.orchestra.run.smithi163.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.3 MB/s | 295 kB 00:00 2024-09-16T13:25:22.586 INFO:teuthology.orchestra.run.smithi163.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.3 MB/s | 100 kB 00:00 2024-09-16T13:25:22.678 INFO:teuthology.orchestra.run.smithi163.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-16T13:25:22.702 INFO:teuthology.orchestra.run.smithi163.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-16T13:25:22.786 INFO:teuthology.orchestra.run.smithi163.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-09-16T13:25:22.820 INFO:teuthology.orchestra.run.smithi163.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-16T13:25:22.845 INFO:teuthology.orchestra.run.smithi163.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 719 kB/s | 42 kB 00:00 2024-09-16T13:25:22.896 INFO:teuthology.orchestra.run.smithi163.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-16T13:25:22.921 INFO:teuthology.orchestra.run.smithi163.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 644 kB/s | 49 kB 00:00 2024-09-16T13:25:22.946 INFO:teuthology.orchestra.run.smithi163.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T13:25:23.030 INFO:teuthology.orchestra.run.smithi163.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T13:25:23.069 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:23.069 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:23.069 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-09-16T13:25:23.069 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.069 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T13:25:23.070 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:23.314 INFO:teuthology.orchestra.run.smithi163.stdout:(22/24): librados2-17.2.7-1643.gd75f2382.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-09-16T13:25:23.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test 2024-09-16T13:25:23.522 INFO:teuthology.orchestra.run.smithi163.stdout:(23/24): ceph-common-17.2.7-1643.gd75f2382.el9. 4.3 MB/s | 22 MB 00:05 2024-09-16T13:25:23.573 INFO:teuthology.orchestra.run.smithi163.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 5.4 MB/s | 2.9 MB 00:00 2024-09-16T13:25:23.578 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:23.578 INFO:teuthology.orchestra.run.smithi163.stdout:Total 10 MB/s | 58 MB 00:05 2024-09-16T13:25:23.640 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:25:23.672 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:25:23.673 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:25:23.992 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:26 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:24.123 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:24.123 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:24.123 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-09-16T13:25:24.124 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:24.125 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 48 M 2024-09-16T13:25:24.125 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 193 M 2024-09-16T13:25:24.125 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:24.338 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:25:24.338 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:25:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-16T13:25:24.435 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-16T13:25:24.485 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 670 kB/s | 163 kB 00:00 2024-09-16T13:25:25.586 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:25:25.965 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T13:25:26.033 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:26.052 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 26 MB/s | 48 MB 00:01 2024-09-16T13:25:26.054 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:26.054 INFO:teuthology.orchestra.run.smithi076.stdout:Total 25 MB/s | 48 MB 00:01 2024-09-16T13:25:26.081 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:26.100 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:26.100 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:26.125 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:26.212 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T13:25:26.243 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:26.315 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:26.375 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T13:25:26.407 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T13:25:26.464 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:26.465 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:26.845 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T13:25:26.880 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T13:25:26.935 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T13:25:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T13:25:26.989 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T13:25:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T13:25:27.077 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T13:25:27.110 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T13:25:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T13:25:27.139 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T13:25:27.167 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T13:25:27.233 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T13:25:27.279 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T13:25:27.317 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T13:25:27.350 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T13:25:27.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T13:25:27.681 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T13:25:27.761 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T13:25:27.836 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T13:25:27.874 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T13:25:30.294 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:30.329 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:30.869 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T13:25:30.908 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T13:25:31.004 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T13:25:31.004 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T13:25:31.004 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T13:25:31.005 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:31.051 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:31.562 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T13:25:31.642 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T13:25:33.865 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T13:25:33.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T13:25:33.865 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T13:25:33.866 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T13:25:34.183 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:34.184 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:34.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph 2024-09-16T13:25:35.059 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:38 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:35.195 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:35.197 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:35.198 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:25:35.198 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:35.198 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:35.198 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T13:25:35.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T13:25:35.200 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T13:25:35.201 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T13:25:35.202 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:Install 34 Packages 2024-09-16T13:25:35.203 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 27 M 2024-09-16T13:25:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 96 M 2024-09-16T13:25:35.204 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:35.853 INFO:teuthology.orchestra.run.smithi076.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-16T13:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-16T13:25:36.103 INFO:teuthology.orchestra.run.smithi076.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-16T13:25:36.203 INFO:teuthology.orchestra.run.smithi076.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-16T13:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.9 MB/s | 240 kB 00:00 2024-09-16T13:25:36.562 INFO:teuthology.orchestra.run.smithi076.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-16T13:25:36.879 INFO:teuthology.orchestra.run.smithi076.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 375 kB/s | 253 kB 00:00 2024-09-16T13:25:36.987 INFO:teuthology.orchestra.run.smithi076.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-16T13:25:37.021 INFO:teuthology.orchestra.run.smithi076.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-16T13:25:37.105 INFO:teuthology.orchestra.run.smithi076.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-16T13:25:37.230 INFO:teuthology.orchestra.run.smithi076.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-16T13:25:37.255 INFO:teuthology.orchestra.run.smithi076.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 918 kB/s | 215 kB 00:00 2024-09-16T13:25:37.789 INFO:teuthology.orchestra.run.smithi076.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 62 kB/s | 35 kB 00:00 2024-09-16T13:25:37.831 INFO:teuthology.orchestra.run.smithi076.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 237 kB/s | 172 kB 00:00 2024-09-16T13:25:37.906 INFO:teuthology.orchestra.run.smithi076.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 119 kB/s | 77 kB 00:00 2024-09-16T13:25:37.956 INFO:teuthology.orchestra.run.smithi076.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 870 kB/s | 43 kB 00:00 2024-09-16T13:25:37.990 INFO:teuthology.orchestra.run.smithi076.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-09-16T13:25:38.049 INFO:teuthology.orchestra.run.smithi076.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 6.0 MB/s | 358 kB 00:00 2024-09-16T13:25:38.074 INFO:teuthology.orchestra.run.smithi076.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-16T13:25:38.107 INFO:teuthology.orchestra.run.smithi076.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 666 kB/s | 18 kB 00:00 2024-09-16T13:25:38.141 INFO:teuthology.orchestra.run.smithi076.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 688 kB/s | 23 kB 00:00 2024-09-16T13:25:38.167 INFO:teuthology.orchestra.run.smithi076.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 774 kB/s | 19 kB 00:00 2024-09-16T13:25:38.192 INFO:teuthology.orchestra.run.smithi076.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 807 kB/s | 20 kB 00:00 2024-09-16T13:25:38.217 INFO:teuthology.orchestra.run.smithi076.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-16T13:25:38.242 INFO:teuthology.orchestra.run.smithi076.stdout:(25/34): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-09-16T13:25:38.293 INFO:teuthology.orchestra.run.smithi076.stdout:(26/34): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-16T13:25:38.318 INFO:teuthology.orchestra.run.smithi076.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-16T13:25:38.343 INFO:teuthology.orchestra.run.smithi076.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-16T13:25:38.360 INFO:teuthology.orchestra.run.smithi076.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-16T13:25:38.394 INFO:teuthology.orchestra.run.smithi076.stdout:(30/34): python3-webob-1.8.8-2.el9.noarch.rpm 6.7 MB/s | 230 kB 00:00 2024-09-16T13:25:38.411 INFO:teuthology.orchestra.run.smithi076.stdout:(31/34): python3-pytz-2021.1-5.el9.noarch.rpm 82 kB/s | 51 kB 00:00 2024-09-16T13:25:38.453 INFO:teuthology.orchestra.run.smithi076.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 482 kB/s | 20 kB 00:00 2024-09-16T13:25:38.469 INFO:teuthology.orchestra.run.smithi076.stdout:(33/34): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-09-16T13:25:38.495 INFO:teuthology.orchestra.run.smithi076.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-16T13:25:38.498 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:38.498 INFO:teuthology.orchestra.run.smithi076.stdout:Total 8.3 MB/s | 27 MB 00:03 2024-09-16T13:25:38.625 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:38.663 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:38.664 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:39.062 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:39.062 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:40.311 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:40.373 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T13:25:40.405 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T13:25:40.438 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T13:25:40.532 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T13:25:40.610 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T13:25:40.670 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T13:25:40.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T13:25:40.721 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:40.721 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-09-16T13:25:40.721 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-09-16T13:25:40.722 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:40.743 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T13:25:40.777 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T13:25:40.811 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T13:25:40.851 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T13:25:40.886 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T13:25:40.920 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T13:25:40.968 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T13:25:41.001 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T13:25:41.083 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T13:25:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T13:25:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T13:25:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T13:25:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T13:25:41.466 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T13:25:41.521 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T13:25:41.709 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T13:25:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T13:25:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T13:25:41.939 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T13:25:41.986 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T13:25:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T13:25:42.101 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:25:42.135 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:42.138 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T13:25:42.211 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T13:25:42.212 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:42.257 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T13:25:42.289 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T13:25:42.316 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T13:25:43.572 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T13:25:43.573 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T13:25:43.574 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T13:25:43.627 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T13:25:43.655 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:44.036 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T13:25:44.060 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:44.262 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T13:25:44.286 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:44.350 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T13:25:44.423 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T13:25:44.423 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:44.423 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-09-16T13:25:44.423 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T13:25:44.424 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:44.425 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:25:44.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-09-16T13:25:45.379 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:25:45.510 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:25:45.511 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-09-16T13:25:45.512 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:45.513 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 48 M 2024-09-16T13:25:45.513 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 193 M 2024-09-16T13:25:45.513 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T13:25:45.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T13:25:45.947 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T13:25:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T13:25:45.971 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-16T13:25:45.996 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 733 kB/s | 163 kB 00:00 2024-09-16T13:25:46.038 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 953 kB/s | 64 kB 00:00 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:25:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T13:25:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T13:25:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:46.867 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:47.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base 2024-09-16T13:25:47.263 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-16T13:25:47.265 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:47.266 INFO:teuthology.orchestra.run.smithi163.stdout:Total 28 MB/s | 48 MB 00:01 2024-09-16T13:25:47.293 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:25:47.313 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:25:47.313 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:25:47.680 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:25:47.680 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:25:47.787 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:50 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:47.877 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:25:47.917 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:47.918 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:25:47.918 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:47.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install cephadm 2024-09-16T13:25:48.211 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:25:48.290 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T13:25:48.329 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T13:25:48.357 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T13:25:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:48.708 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:48.709 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:25:48.710 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:48.710 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 83 k 2024-09-16T13:25:48.710 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 375 k 2024-09-16T13:25:48.710 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:48.886 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 470 kB/s | 83 kB 00:00 2024-09-16T13:25:48.887 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:48.887 INFO:teuthology.orchestra.run.smithi076.stdout:Total 466 kB/s | 83 kB 00:00 2024-09-16T13:25:48.887 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:48.892 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:48.892 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:48.894 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:48.894 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:48.968 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:49.410 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:25:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:25:49.914 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:25:50.145 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:25:50.145 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:50.145 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:50.145 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:25:50.146 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:50.146 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:50.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T13:25:50.915 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:51.047 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:51.048 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:51.048 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T13:25:51.048 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:51.048 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 137 k 2024-09-16T13:25:51.049 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 379 k 2024-09-16T13:25:51.050 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:51.242 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 714 kB/s | 137 kB 00:00 2024-09-16T13:25:51.242 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:51.243 INFO:teuthology.orchestra.run.smithi076.stdout:Total 708 kB/s | 137 kB 00:00 2024-09-16T13:25:51.243 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:51.251 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:51.251 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:51.308 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:51.308 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:51.518 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:25:51.542 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:25:51.543 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:25:51.543 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T13:25:51.543 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:52.456 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:25:52.456 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:52.456 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:52.456 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:52.456 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:52.457 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:52.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr 2024-09-16T13:25:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T13:25:53.047 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T13:25:53.252 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:53.346 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:25:53.389 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:53.390 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:25:53.390 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:53.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T13:25:54.023 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:54.157 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:54.157 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:54.157 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:25:54.157 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout:Install 7 Packages 2024-09-16T13:25:54.158 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:54.159 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.7 M 2024-09-16T13:25:54.159 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 80 M 2024-09-16T13:25:54.159 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:54.762 INFO:teuthology.orchestra.run.smithi076.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 157 kB/s | 24 kB 00:00 2024-09-16T13:25:54.787 INFO:teuthology.orchestra.run.smithi076.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 81 kB/s | 15 kB 00:00 2024-09-16T13:25:54.830 INFO:teuthology.orchestra.run.smithi076.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-16T13:25:54.854 INFO:teuthology.orchestra.run.smithi076.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T13:25:54.880 INFO:teuthology.orchestra.run.smithi076.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-16T13:25:54.905 INFO:teuthology.orchestra.run.smithi076.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T13:25:55.030 INFO:teuthology.orchestra.run.smithi076.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 7.9 MB/s | 3.4 MB 00:00 2024-09-16T13:25:55.034 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:55.034 INFO:teuthology.orchestra.run.smithi076.stdout:Total 4.2 MB/s | 3.7 MB 00:00 2024-09-16T13:25:55.034 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:25:55.042 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:25:55.042 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:25:55.134 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:25:55.134 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:25:55.422 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T13:25:55.422 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T13:25:55.422 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T13:25:55.422 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T13:25:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:25:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T13:25:55.524 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T13:25:55.579 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T13:25:55.611 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T13:25:55.640 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:55.754 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:25:56.071 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-09-16T13:25:56.637 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T13:25:56.648 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:41 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:25:56.668 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T13:25:56.782 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:25:56.784 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections 2024-09-16T13:25:56.785 INFO:teuthology.orchestra.run.smithi163.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T13:25:56.786 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout:Install 34 Packages 2024-09-16T13:25:56.787 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:25:56.788 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 27 M 2024-09-16T13:25:56.788 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 96 M 2024-09-16T13:25:56.788 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:25:57.391 INFO:teuthology.orchestra.run.smithi163.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-16T13:25:57.583 INFO:teuthology.orchestra.run.smithi163.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-16T13:25:57.633 INFO:teuthology.orchestra.run.smithi163.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-16T13:25:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T13:25:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T13:25:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T13:25:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T13:25:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T13:25:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T13:25:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T13:25:57.734 INFO:teuthology.orchestra.run.smithi163.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.3 MB/s | 240 kB 00:00 2024-09-16T13:25:57.809 INFO:teuthology.orchestra.run.smithi163.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 9.4 MB/s | 3.9 MB 00:00 2024-09-16T13:25:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T13:25:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:57.952 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:25:58.126 INFO:teuthology.orchestra.run.smithi163.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 28 MB/s | 15 MB 00:00 2024-09-16T13:25:58.159 INFO:teuthology.orchestra.run.smithi163.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 595 kB/s | 253 kB 00:00 2024-09-16T13:25:58.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T13:25:58.276 INFO:teuthology.orchestra.run.smithi163.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 2.6 MB/s | 1.2 MB 00:00 2024-09-16T13:25:58.305 INFO:teuthology.orchestra.run.smithi163.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 953 kB/s | 135 kB 00:00 2024-09-16T13:25:58.335 INFO:teuthology.orchestra.run.smithi163.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-16T13:25:58.360 INFO:teuthology.orchestra.run.smithi163.stdout:(11/34): python3-ply-3.11-14.el9.noarch.rpm 454 kB/s | 106 kB 00:00 2024-09-16T13:25:58.469 INFO:teuthology.orchestra.run.smithi163.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-16T13:25:58.519 INFO:teuthology.orchestra.run.smithi163.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 936 kB/s | 172 kB 00:00 2024-09-16T13:25:58.545 INFO:teuthology.orchestra.run.smithi163.stdout:(14/34): python3-urllib3-1.26.5-6.el9.noarch.rp 884 kB/s | 215 kB 00:00 2024-09-16T13:25:58.570 INFO:teuthology.orchestra.run.smithi163.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 766 kB/s | 77 kB 00:00 2024-09-16T13:25:58.595 INFO:teuthology.orchestra.run.smithi163.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-16T13:25:58.646 INFO:teuthology.orchestra.run.smithi163.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-09-16T13:25:58.671 INFO:teuthology.orchestra.run.smithi163.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 329 kB/s | 42 kB 00:00 2024-09-16T13:25:58.705 INFO:teuthology.orchestra.run.smithi163.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-16T13:25:58.730 INFO:teuthology.orchestra.run.smithi163.stdout:(20/34): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-09-16T13:25:58.764 INFO:teuthology.orchestra.run.smithi163.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-09-16T13:25:58.768 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:25:58.789 INFO:teuthology.orchestra.run.smithi163.stdout:(22/34): python3-jaraco-classes-3.2.1-5.el9.noa 215 kB/s | 18 kB 00:00 2024-09-16T13:25:58.815 INFO:teuthology.orchestra.run.smithi163.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 275 kB/s | 23 kB 00:00 2024-09-16T13:25:58.840 INFO:teuthology.orchestra.run.smithi163.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-16T13:25:58.865 INFO:teuthology.orchestra.run.smithi163.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-16T13:25:58.890 INFO:teuthology.orchestra.run.smithi163.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-16T13:25:58.901 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T13:25:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout:Install 12 Packages 2024-09-16T13:25:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:25:58.905 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 38 M 2024-09-16T13:25:58.905 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 212 M 2024-09-16T13:25:58.905 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:25:58.915 INFO:teuthology.orchestra.run.smithi163.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T13:25:58.941 INFO:teuthology.orchestra.run.smithi163.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T13:25:58.966 INFO:teuthology.orchestra.run.smithi163.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T13:25:58.992 INFO:teuthology.orchestra.run.smithi163.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-16T13:25:59.021 INFO:teuthology.orchestra.run.smithi163.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T13:25:59.046 INFO:teuthology.orchestra.run.smithi163.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-09-16T13:25:59.071 INFO:teuthology.orchestra.run.smithi163.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T13:25:59.105 INFO:teuthology.orchestra.run.smithi163.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.7 MB/s | 427 kB 00:00 2024-09-16T13:25:59.109 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:25:59.109 INFO:teuthology.orchestra.run.smithi163.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-16T13:25:59.122 INFO:teuthology.orchestra.run.smithi076.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-16T13:25:59.189 INFO:teuthology.orchestra.run.smithi076.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-16T13:25:59.233 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:25:59.271 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:25:59.271 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:25:59.364 INFO:teuthology.orchestra.run.smithi076.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-09-16T13:25:59.406 INFO:teuthology.orchestra.run.smithi076.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-16T13:25:59.498 INFO:teuthology.orchestra.run.smithi076.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-16T13:25:59.523 INFO:teuthology.orchestra.run.smithi076.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 330 kB/s | 38 kB 00:00 2024-09-16T13:25:59.565 INFO:teuthology.orchestra.run.smithi076.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-09-16T13:25:59.677 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:25:59.678 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:25:59.724 INFO:teuthology.orchestra.run.smithi076.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-09-16T13:25:59.766 INFO:teuthology.orchestra.run.smithi076.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-16T13:25:59.833 INFO:teuthology.orchestra.run.smithi076.stdout:(10/12): libgfortran-11.5.0-2.el9.x86_64.rpm 958 kB/s | 801 kB 00:00 2024-09-16T13:26:00.183 INFO:teuthology.orchestra.run.smithi076.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-09-16T13:26:00.333 INFO:teuthology.orchestra.run.smithi076.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.9 MB/s | 4.9 MB 00:00 2024-09-16T13:26:00.334 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:00.334 INFO:teuthology.orchestra.run.smithi076.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-16T13:26:00.692 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:00.718 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:00.719 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T13:26:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T13:26:01.115 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T13:26:01.207 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T13:26:01.229 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:01.229 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:01.305 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T13:26:01.352 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T13:26:01.387 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T13:26:01.416 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T13:26:01.451 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T13:26:01.484 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T13:26:01.525 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T13:26:01.560 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T13:26:01.595 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T13:26:01.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T13:26:01.675 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T13:26:01.749 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T13:26:01.896 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T13:26:01.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T13:26:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:02.037 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T13:26:02.076 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T13:26:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T13:26:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T13:26:02.134 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T13:26:02.187 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T13:26:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T13:26:02.265 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T13:26:02.382 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T13:26:02.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T13:26:02.451 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T13:26:02.481 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T13:26:02.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T13:26:02.563 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T13:26:02.602 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T13:26:02.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T13:26:02.759 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T13:26:02.906 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T13:26:02.938 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T13:26:02.963 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:03.209 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T13:26:03.337 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T13:26:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T13:26:04.404 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T13:26:04.404 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:04.404 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T13:26:04.404 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T13:26:04.405 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T13:26:04.405 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:04.611 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T13:26:04.784 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T13:26:04.809 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:05.018 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T13:26:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T13:26:05.042 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:05.042 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T13:26:05.042 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T13:26:05.042 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T13:26:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:05.106 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T13:26:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T13:26:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T13:26:06.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T13:26:06.849 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T13:26:06.850 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T13:26:06.851 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T13:26:07.821 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T13:26:07.822 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:07.823 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T13:26:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T13:26:08.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T13:26:08.614 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:08.615 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:08.815 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:08.906 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:08.947 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:08.948 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:08.948 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:09.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-rook 2024-09-16T13:26:09.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-09-16T13:26:09.594 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:09.618 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:09.725 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:09.726 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:26:09.727 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 83 k 2024-09-16T13:26:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 375 k 2024-09-16T13:26:09.727 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:09.763 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:09.764 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:26:09.764 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T13:26:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2024-09-16T13:26:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:09.767 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 2.2 M 2024-09-16T13:26:09.767 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 26 M 2024-09-16T13:26:09.767 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:09.919 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 431 kB/s | 83 kB 00:00 2024-09-16T13:26:09.920 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:09.920 INFO:teuthology.orchestra.run.smithi163.stdout:Total 427 kB/s | 83 kB 00:00 2024-09-16T13:26:09.920 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:09.925 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:09.925 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:09.927 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:09.927 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:10.003 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:10.148 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 333 kB/s | 26 kB 00:00 2024-09-16T13:26:10.173 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 187 kB/s | 19 kB 00:00 2024-09-16T13:26:10.232 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-16T13:26:10.257 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-16T13:26:10.283 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 225 kB/s | 47 kB 00:00 2024-09-16T13:26:10.308 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T13:26:10.341 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-09-16T13:26:10.392 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-09-16T13:26:10.417 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 129 kB/s | 14 kB 00:00 2024-09-16T13:26:10.444 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:10.451 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-16T13:26:10.476 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-16T13:26:10.501 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T13:26:10.531 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:10.535 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-16T13:26:10.538 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:10.538 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-09-16T13:26:10.602 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:10.617 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:10.618 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:10.704 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:10.705 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:10.853 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:11.113 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:11.205 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:11.368 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T13:26:11.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T13:26:11.428 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T13:26:11.484 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T13:26:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T13:26:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T13:26:11.665 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T13:26:11.788 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T13:26:11.839 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T13:26:11.950 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:12.083 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:12.084 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:12.085 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:26:12.085 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:12.085 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 137 k 2024-09-16T13:26:12.085 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 379 k 2024-09-16T13:26:12.085 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:12.310 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 611 kB/s | 137 kB 00:00 2024-09-16T13:26:12.310 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:12.311 INFO:teuthology.orchestra.run.smithi163.stdout:Total 606 kB/s | 137 kB 00:00 2024-09-16T13:26:12.311 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:12.318 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:12.318 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:12.375 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:12.375 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:12.532 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T13:26:12.603 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:26:12.611 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T13:26:12.628 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:26:12.628 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:12.628 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T13:26:12.628 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T13:26:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T13:26:12.714 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T13:26:13.513 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T13:26:13.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:13.515 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:13.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-09-16T13:26:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T13:26:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:13.844 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:13.845 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:14.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T13:26:14.347 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:14.450 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:14.493 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:14.494 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:14.494 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:14.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T13:26:14.680 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:17 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:14.812 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T13:26:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 7.0 M 2024-09-16T13:26:14.814 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 33 M 2024-09-16T13:26:14.815 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:15.142 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:15.276 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T13:26:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout:Install 7 Packages 2024-09-16T13:26:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:15.279 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.7 M 2024-09-16T13:26:15.279 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 80 M 2024-09-16T13:26:15.279 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:15.329 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T13:26:15.354 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 544 kB/s | 124 kB 00:00 2024-09-16T13:26:15.405 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T13:26:15.430 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 768 kB/s | 58 kB 00:00 2024-09-16T13:26:15.455 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-16T13:26:15.607 INFO:teuthology.orchestra.run.smithi163.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 169 kB/s | 24 kB 00:00 2024-09-16T13:26:15.631 INFO:teuthology.orchestra.run.smithi163.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 86 kB/s | 15 kB 00:00 2024-09-16T13:26:15.674 INFO:teuthology.orchestra.run.smithi163.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-16T13:26:15.707 INFO:teuthology.orchestra.run.smithi163.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 976 kB/s | 31 kB 00:00 2024-09-16T13:26:15.733 INFO:teuthology.orchestra.run.smithi163.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 405 kB/s | 41 kB 00:00 2024-09-16T13:26:15.766 INFO:teuthology.orchestra.run.smithi163.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-16T13:26:15.883 INFO:teuthology.orchestra.run.smithi163.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.0 MB/s | 3.4 MB 00:00 2024-09-16T13:26:15.887 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:15.887 INFO:teuthology.orchestra.run.smithi163.stdout:Total 6.0 MB/s | 3.7 MB 00:00 2024-09-16T13:26:15.887 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:15.896 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:15.896 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:15.975 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:15.976 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:16.282 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:16.351 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T13:26:16.385 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T13:26:16.440 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T13:26:16.472 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T13:26:16.501 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T13:26:16.839 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.5 MB/s | 6.0 MB 00:01 2024-09-16T13:26:16.842 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:16.842 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-09-16T13:26:16.912 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:16.923 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:16.924 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:17.021 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:17.022 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:17.325 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:17.480 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T13:26:17.498 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T13:26:17.538 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T13:26:17.540 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T13:26:18.064 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T13:26:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T13:26:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T13:26:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T13:26:18.611 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T13:26:18.611 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T13:26:18.611 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T13:26:18.611 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T13:26:18.611 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T13:26:18.612 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T13:26:18.612 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T13:26:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T13:26:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:18.894 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:19.034 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T13:26:19.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T13:26:19.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T13:26:19.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T13:26:19.035 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T13:26:19.035 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T13:26:19.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T13:26:19.309 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:19.310 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:19.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse 2024-09-16T13:26:19.723 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:19.857 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:26:19.858 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-09-16T13:26:19.859 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-09-16T13:26:19.860 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:19.860 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 38 M 2024-09-16T13:26:19.860 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 212 M 2024-09-16T13:26:19.860 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:20.165 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:23 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:20.297 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:20.297 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:20.297 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:26:20.297 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:20.297 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2024-09-16T13:26:20.298 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:20.299 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 893 k 2024-09-16T13:26:20.299 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.6 M 2024-09-16T13:26:20.299 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:20.392 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-16T13:26:20.466 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 545 kB/s | 80 kB 00:00 2024-09-16T13:26:20.501 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 303 kB/s | 33 kB 00:00 2024-09-16T13:26:20.591 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-16T13:26:20.592 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:20.592 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.0 MB/s | 893 kB 00:00 2024-09-16T13:26:20.604 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:20.626 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:20.626 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:20.704 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:20.704 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:20.759 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 230 kB/s | 191 kB 00:00 2024-09-16T13:26:20.851 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-09-16T13:26:20.894 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:20.918 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): libgfortran-11.5.0-2.el9.x86_64.rpm 808 kB/s | 801 kB 00:00 2024-09-16T13:26:20.969 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-16T13:26:21.051 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T13:26:21.114 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T13:26:21.253 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 863 kB/s | 245 kB 00:00 2024-09-16T13:26:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T13:26:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:22.001 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:22.103 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.9 MB/s | 3.0 MB 00:01 2024-09-16T13:26:22.261 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 250 kB 00:00 2024-09-16T13:26:22.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-volume 2024-09-16T13:26:22.934 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:23.070 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 270 k 2024-09-16T13:26:23.071 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.3 M 2024-09-16T13:26:23.072 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:23.379 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 879 kB/s | 270 kB 00:00 2024-09-16T13:26:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:Total 874 kB/s | 270 kB 00:00 2024-09-16T13:26:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:23.388 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:23.389 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:23.419 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:23.419 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:23.630 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:23.698 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:23.714 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:23.714 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:23.715 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T13:26:23.715 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:23.861 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 1.7 MB/s | 4.9 MB 00:02 2024-09-16T13:26:23.995 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:24.956 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel 2024-09-16T13:26:25.532 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:25.668 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:25.669 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:25.670 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 126 k 2024-09-16T13:26:25.670 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 448 k 2024-09-16T13:26:25.670 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:25.876 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 612 kB/s | 126 kB 00:00 2024-09-16T13:26:25.877 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:25.877 INFO:teuthology.orchestra.run.smithi076.stdout:Total 607 kB/s | 126 kB 00:00 2024-09-16T13:26:25.877 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:25.883 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:25.883 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:25.947 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:25.947 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:26.095 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:26.215 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:26.361 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:04 2024-09-16T13:26:26.362 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:26.363 INFO:teuthology.orchestra.run.smithi163.stdout:Total 5.9 MB/s | 38 MB 00:06 2024-09-16T13:26:26.641 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:26.726 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:26.752 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:26.752 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:26.910 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:26.910 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:26.910 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:26.910 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:26.911 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:26.911 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:27.153 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2 2024-09-16T13:26:27.258 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:27.258 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:27.737 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:27.830 INFO:teuthology.orchestra.run.smithi076.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:27.874 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:27.875 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:27.875 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:27.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs-devel 2024-09-16T13:26:28.019 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T13:26:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T13:26:28.233 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T13:26:28.277 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T13:26:28.463 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T13:26:28.493 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T13:26:28.522 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T13:26:28.656 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:28.657 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:28.657 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T13:26:28.657 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 25 k 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 110 k 2024-09-16T13:26:28.658 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:28.813 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 161 kB/s | 25 kB 00:00 2024-09-16T13:26:28.814 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:28.814 INFO:teuthology.orchestra.run.smithi076.stdout:Total 160 kB/s | 25 kB 00:00 2024-09-16T13:26:28.814 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:28.817 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:28.817 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:28.819 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:28.819 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:28.896 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:28.992 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:29.265 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T13:26:29.367 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T13:26:29.591 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:29.591 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:29.591 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:29.591 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:29.591 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:29.592 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:29.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2 2024-09-16T13:26:30.341 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:30.433 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:30.477 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:30.478 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:30.478 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:30.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1 2024-09-16T13:26:30.638 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T13:26:31.129 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:31.221 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:31.264 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:31.265 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:31.265 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:31.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados 2024-09-16T13:26:31.685 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T13:26:31.720 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T13:26:31.917 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:32.006 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:32.048 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:32.049 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:32.049 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:32.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw 2024-09-16T13:26:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:32.827 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:32.828 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:32.828 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:32.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs 2024-09-16T13:26:33.474 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:33.564 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:33.607 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:33.608 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:33.608 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:33.697 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd 2024-09-16T13:26:34.263 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:34.341 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T13:26:34.342 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T13:26:34.353 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:34.396 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:34.397 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-09-16T13:26:34.398 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:34.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse 2024-09-16T13:26:34.732 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T13:26:34.732 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T13:26:34.733 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T13:26:34.734 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T13:26:34.735 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T13:26:34.735 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:34.735 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:35.047 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:35.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-09-16T13:26:35.180 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:35.181 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:35.181 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T13:26:35.181 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:35.181 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 82 k 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 222 k 2024-09-16T13:26:35.182 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:35.374 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 426 kB/s | 82 kB 00:00 2024-09-16T13:26:35.375 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:35.375 INFO:teuthology.orchestra.run.smithi076.stdout:Total 423 kB/s | 82 kB 00:00 2024-09-16T13:26:35.375 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:35.382 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:35.382 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:35.445 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:35.445 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:35.721 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:35.862 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T13:26:35.863 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T13:26:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.2 M 2024-09-16T13:26:35.865 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-09-16T13:26:35.866 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:36.096 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 267 kB/s | 47 kB 00:00 2024-09-16T13:26:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:36.323 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:36.323 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:36.438 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-16T13:26:36.497 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 44 kB/s | 26 kB 00:00 2024-09-16T13:26:36.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror 2024-09-16T13:26:36.922 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-09-16T13:26:37.156 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-16T13:26:37.201 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:37.214 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 632 kB/s | 32 kB 00:00 2024-09-16T13:26:37.240 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 559 kB/s | 14 kB 00:00 2024-09-16T13:26:37.291 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-16T13:26:37.324 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 337 kB/s | 279 kB 00:00 2024-09-16T13:26:37.340 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:37.340 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:37.340 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-09-16T13:26:37.340 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:37.341 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 2.9 M 2024-09-16T13:26:37.342 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 12 M 2024-09-16T13:26:37.342 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:37.375 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-16T13:26:37.408 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-16T13:26:37.458 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-16T13:26:37.575 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2024-09-16T13:26:37.578 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:37.578 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-09-16T13:26:37.644 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:37.658 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:37.658 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:37.678 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-16T13:26:37.679 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:37.679 INFO:teuthology.orchestra.run.smithi076.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-16T13:26:37.680 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:37.689 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:37.689 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:37.747 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:37.748 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:37.767 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:37.768 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:38.178 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:38.237 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:38.258 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:38.262 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:38.262 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:38.262 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T13:26:38.263 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T13:26:38.263 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T13:26:38.263 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:38.425 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T13:26:38.485 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T13:26:38.549 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T13:26:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T13:26:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T13:26:38.738 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T13:26:38.841 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T13:26:38.896 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T13:26:39.294 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:39.295 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:39.295 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:39.295 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:39.295 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:39.295 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:39.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd 2024-09-16T13:26:39.618 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T13:26:39.686 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T13:26:39.719 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T13:26:39.765 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T13:26:39.789 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T13:26:40.144 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 01:24:57 PM UTC. 2024-09-16T13:26:40.286 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-09-16T13:26:40.287 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:40.287 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-09-16T13:26:40.287 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:40.287 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-09-16T13:26:40.287 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 165 k 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 477 k 2024-09-16T13:26:40.288 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-09-16T13:26:40.499 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 784 kB/s | 165 kB 00:00 2024-09-16T13:26:40.500 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:40.500 INFO:teuthology.orchestra.run.smithi076.stdout:Total 777 kB/s | 165 kB 00:00 2024-09-16T13:26:40.500 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-09-16T13:26:40.508 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-09-16T13:26:40.508 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-09-16T13:26:40.527 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T13:26:40.527 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T13:26:40.527 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T13:26:40.527 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T13:26:40.527 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T13:26:40.528 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T13:26:40.575 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-09-16T13:26:40.575 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-09-16T13:26:40.733 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-09-16T13:26:40.848 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:40.863 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T13:26:40.863 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:40.863 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:40.863 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:40.863 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:40.864 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:41.136 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T13:26:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:41.507 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:41.508 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:41.508 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-09-16T13:26:41.508 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:41.508 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:26:41.508 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-09-16T13:26:41.700 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:41.742 DEBUG:teuthology.parallel:result is None 2024-09-16T13:26:41.835 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T13:26:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 7.0 M 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-09-16T13:26:41.837 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:42.238 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 598 kB/s | 124 kB 00:00 2024-09-16T13:26:42.330 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-16T13:26:42.364 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-16T13:26:42.389 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T13:26:42.798 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 324 kB/s | 248 kB 00:00 2024-09-16T13:26:43.173 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-16T13:26:43.175 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:43.175 INFO:teuthology.orchestra.run.smithi163.stdout:Total 5.2 MB/s | 7.0 MB 00:01 2024-09-16T13:26:43.245 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:43.257 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:43.257 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:43.356 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:43.357 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:43.662 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:43.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T13:26:43.883 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T13:26:44.412 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T13:26:44.495 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T13:26:44.550 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T13:26:44.579 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T13:26:45.351 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T13:26:45.351 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T13:26:45.351 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T13:26:45.352 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T13:26:45.352 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T13:26:45.352 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T13:26:45.637 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T13:26:45.638 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T13:26:45.638 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T13:26:45.638 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T13:26:45.638 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:45.638 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:45.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-09-16T13:26:46.496 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:46.630 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:46.631 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:46.632 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:46.632 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-09-16T13:26:46.632 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:46.632 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 893 k 2024-09-16T13:26:46.632 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.6 M 2024-09-16T13:26:46.633 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:47.009 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 674 kB/s | 80 kB 00:00 2024-09-16T13:26:47.175 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-16T13:26:47.176 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:47.176 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-16T13:26:47.189 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:47.211 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:47.211 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:47.290 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:47.290 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:47.625 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T13:26:47.691 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T13:26:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T13:26:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:48.660 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:48.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-volume 2024-09-16T13:26:49.579 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:49.716 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:49.717 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 270 k 2024-09-16T13:26:49.718 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.3 M 2024-09-16T13:26:49.718 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:49.923 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-16T13:26:49.923 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:49.924 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-16T13:26:49.924 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:49.932 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:49.932 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:49.960 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:49.960 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:50.173 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:50.275 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:26:50.275 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T13:26:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T13:26:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:51.337 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T13:26:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:51.338 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:51.694 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-09-16T13:26:52.278 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:52.413 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:52.413 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T13:26:52.413 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 126 k 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 448 k 2024-09-16T13:26:52.414 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:52.621 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 611 kB/s | 126 kB 00:00 2024-09-16T13:26:52.622 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:52.622 INFO:teuthology.orchestra.run.smithi163.stdout:Total 606 kB/s | 126 kB 00:00 2024-09-16T13:26:52.622 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:52.628 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:52.628 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:52.690 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:52.691 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:52.843 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:53.363 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:53.615 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:53.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-09-16T13:26:54.467 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:54.558 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:54.601 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:54.602 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:54.603 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:54.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-09-16T13:26:55.265 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:55.399 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:55.400 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 25 k 2024-09-16T13:26:55.401 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 110 k 2024-09-16T13:26:55.401 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:26:55.543 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 175 kB/s | 25 kB 00:00 2024-09-16T13:26:55.544 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:26:55.544 INFO:teuthology.orchestra.run.smithi163.stdout:Total 173 kB/s | 25 kB 00:00 2024-09-16T13:26:55.544 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:26:55.547 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:26:55.547 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:26:55.548 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:26:55.549 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:26:55.631 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:26:55.747 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:56.129 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:56.367 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:26:56.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:56.368 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:26:56.368 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:26:56.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:26:56.368 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:56.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-09-16T13:26:57.135 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:57.226 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:57.268 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:57.269 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:57.269 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:57.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-09-16T13:26:57.928 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:58.019 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:58.062 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:58.063 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:58.063 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:58.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-09-16T13:26:58.715 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:58.805 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:58.848 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:58.849 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:58.849 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:58.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-09-16T13:26:59.496 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:26:59.585 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:26:59.627 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:26:59.628 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:26:59.628 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:26:59.711 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-09-16T13:27:00.276 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:27:00.366 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:27:00.408 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:27:00.409 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:27:00.409 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:27:00.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-09-16T13:27:01.067 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:27:01.157 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T13:27:01.200 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:27:01.201 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T13:27:01.201 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:27:01.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-09-16T13:27:01.856 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:27:01.990 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:01.991 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:27:01.992 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:01.992 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 82 k 2024-09-16T13:27:01.992 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 222 k 2024-09-16T13:27:01.992 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:27:02.188 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 418 kB/s | 82 kB 00:00 2024-09-16T13:27:02.189 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:27:02.189 INFO:teuthology.orchestra.run.smithi163.stdout:Total 415 kB/s | 82 kB 00:00 2024-09-16T13:27:02.189 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:27:02.196 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:27:02.196 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:27:02.258 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:27:02.259 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:27:02.412 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:27:02.518 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:02.898 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:03.162 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:27:03.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-09-16T13:27:04.078 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:27:04.214 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:27:04.215 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:04.216 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.9 M 2024-09-16T13:27:04.216 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 12 M 2024-09-16T13:27:04.216 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:27:04.568 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.3 MB/s | 2.9 MB 00:00 2024-09-16T13:27:04.569 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:27:04.569 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.3 MB/s | 2.9 MB 00:00 2024-09-16T13:27:04.569 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:27:04.578 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:27:04.578 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:27:04.657 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:27:04.658 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:27:05.075 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:27:05.127 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T13:27:05.152 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:06.269 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:06.269 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:06.269 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:27:06.269 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:27:06.270 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:06.270 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:27:06.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-09-16T13:27:07.130 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 01:25:15 PM UTC. 2024-09-16T13:27:07.274 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:07.275 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 165 k 2024-09-16T13:27:07.276 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 477 k 2024-09-16T13:27:07.276 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T13:27:07.482 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 802 kB/s | 165 kB 00:00 2024-09-16T13:27:07.482 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T13:27:07.483 INFO:teuthology.orchestra.run.smithi163.stdout:Total 796 kB/s | 165 kB 00:00 2024-09-16T13:27:07.483 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T13:27:07.491 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T13:27:07.491 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T13:27:07.558 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T13:27:07.558 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T13:27:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T13:27:07.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:08.277 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:08.549 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T13:27:08.550 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:08.550 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T13:27:08.550 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T13:27:08.550 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:08.550 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T13:27:08.858 DEBUG:teuthology.parallel:result is None 2024-09-16T13:27:08.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:27:09.010 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T13:27:09.037 INFO:teuthology.orchestra.run.smithi076.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T13:27:09.037 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T13:27:09.037 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T13:27:09.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:27:09.191 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T13:27:09.213 INFO:teuthology.orchestra.run.smithi163.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T13:27:09.213 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T13:27:09.213 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T13:27:09.215 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T13:27:09.215 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:09.215 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T13:27:09.246 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:09.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T13:27:09.280 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T13:27:09.280 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:09.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T13:27:09.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T13:27:09.380 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:09.380 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T13:27:09.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T13:27:09.471 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T13:27:09.471 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:09.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T13:27:09.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T13:27:09.564 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:09.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T13:27:09.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T13:27:09.654 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T13:27:09.655 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:09.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T13:27:09.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T13:27:09.745 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:09.745 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T13:27:09.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T13:27:09.834 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T13:27:09.905 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T13:27:09.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2024-09-16T13:27:09.909 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2024-09-16T13:27:09.934 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T13:27:09.934 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T13:27:09.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T13:27:10.000 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T13:27:10.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T13:27:10.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T13:27:10.076 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T13:27:10.076 INFO:tasks.ceph:config {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T13:27:10.076 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writethrough}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907305', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMO+gKCWgKmEx2hg9z3aH8/y+UKBc3gY4+qAQVhKx+mFWZuxbYwVhFTECXkmypzj55iAvj4fGTiS0chpy0sKWY8=', 'smithi163.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL0IU3QQdFnLPQv8FQWY3up2buWqKw+C3q/al6rb/lW3u+UiB2EpvfU5ap4y+TscP5jI4NcqaxugdM4GYZ26Urg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T13:27:10.077 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T13:27:10.108 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T13:27:10.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T13:27:10.164 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T13:27:10.438 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:10.439 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T13:27:10.470 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T13:27:10.470 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-09-16T13:27:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T13:27:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 13:26:38.513771501 +0000 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 13:24:16.910874473 +0000 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 13:24:16.910874473 +0000 2024-09-16T13:27:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 13:24:16.910874473 +0000 2024-09-16T13:27:10.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T13:27:10.589 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T13:27:10.589 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T13:27:10.589 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000262734 s, 1.9 MB/s 2024-09-16T13:27:10.591 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T13:27:10.647 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-09-16T13:27:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T13:27:10.702 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:10.702 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 13:26:38.514771471 +0000 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 13:24:17.216865056 +0000 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 13:24:17.216865056 +0000 2024-09-16T13:27:10.703 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 13:24:17.216865056 +0000 2024-09-16T13:27:10.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T13:27:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T13:27:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T13:27:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000280344 s, 1.8 MB/s 2024-09-16T13:27:10.768 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T13:27:10.823 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 13:26:38.514771471 +0000 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 13:24:17.532855331 +0000 2024-09-16T13:27:10.882 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 13:24:17.532855331 +0000 2024-09-16T13:27:10.883 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 13:24:17.532855331 +0000 2024-09-16T13:27:10.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T13:27:10.945 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T13:27:10.945 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T13:27:10.945 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000194717 s, 2.6 MB/s 2024-09-16T13:27:10.946 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T13:27:11.002 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 13:26:38.515771440 +0000 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 13:24:17.854845421 +0000 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 13:24:17.854845421 +0000 2024-09-16T13:27:11.058 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-09-16 13:24:17.854845421 +0000 2024-09-16T13:27:11.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T13:27:11.121 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T13:27:11.121 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T13:27:11.121 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000269244 s, 1.9 MB/s 2024-09-16T13:27:11.122 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T13:27:11.177 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:11.178 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:11.178 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T13:27:11.197 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T13:27:11.197 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-09-16T13:27:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 13:27:05.414938578 +0000 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 13:24:19.020674561 +0000 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 13:24:19.020674561 +0000 2024-09-16T13:27:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 13:24:19.020674561 +0000 2024-09-16T13:27:11.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T13:27:11.324 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T13:27:11.324 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T13:27:11.324 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000297505 s, 1.7 MB/s 2024-09-16T13:27:11.325 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T13:27:11.385 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 13:27:05.414938578 +0000 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 13:24:19.349666987 +0000 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 13:24:19.349666987 +0000 2024-09-16T13:27:11.444 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 13:24:19.349666987 +0000 2024-09-16T13:27:11.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T13:27:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T13:27:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T13:27:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000229165 s, 2.2 MB/s 2024-09-16T13:27:11.513 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T13:27:11.571 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-09-16T13:27:11.629 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T13:27:11.629 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:11.629 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 13:27:05.414938578 +0000 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 13:24:19.685659251 +0000 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 13:24:19.685659251 +0000 2024-09-16T13:27:11.630 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 13:24:19.685659251 +0000 2024-09-16T13:27:11.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T13:27:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T13:27:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T13:27:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000241523 s, 2.1 MB/s 2024-09-16T13:27:11.697 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T13:27:11.755 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-09-16T13:27:11.812 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T13:27:11.812 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:27:11.812 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T13:27:11.812 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:27:11.813 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T13:27:11.813 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 13:27:05.415938556 +0000 2024-09-16T13:27:11.813 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 13:24:19.998652045 +0000 2024-09-16T13:27:11.813 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 13:24:19.998652045 +0000 2024-09-16T13:27:11.813 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 13:24:19.998652045 +0000 2024-09-16T13:27:11.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T13:27:11.877 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T13:27:11.878 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T13:27:11.878 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000260803 s, 2.0 MB/s 2024-09-16T13:27:11.879 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T13:27:11.936 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:11.936 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T13:27:11.936 INFO:tasks.ceph:Generating config... 2024-09-16T13:27:11.937 INFO:tasks.ceph:[client] rbd cache = True 2024-09-16T13:27:11.937 INFO:tasks.ceph:[client] rbd cache max dirty = 0 2024-09-16T13:27:11.937 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T13:27:11.937 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T13:27:11.938 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T13:27:11.938 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T13:27:11.938 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T13:27:11.938 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T13:27:11.938 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T13:27:11.938 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T13:27:11.938 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T13:27:11.939 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T13:27:11.939 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T13:27:11.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T13:27:11.992 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T13:27:11.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T13:27:12.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:27:12.101 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.76'), ('mon.c', '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]'), ('mon.b', '172.21.15.163')] 2024-09-16T13:27:12.102 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.76,[v2:172.21.15.76:3301,v1:172.21.15.76:6790],172.21.15.163', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd cache': True, 'rbd cache max dirty': 0}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T13:27:12.102 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:12.102 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T13:27:12.157 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.76 --addv c '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]' --add b 172.21.15.163 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:12.240 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:12.240 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: generated fsid 0cf46de4-f58b-4e68-a8b9-534bb33f8d72 2024-09-16T13:27:12.240 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = octopus 2024-09-16T13:27:12.240 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 0 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 0cf46de4-f58b-4e68-a8b9-534bb33f8d72 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:last_changed 2024-09-16T13:27:12.239382+0000 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-09-16T13:27:12.239382+0000 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:min_mon_release 15 (octopus) 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:election_strategy: 1 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:2: [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] mon.c 2024-09-16T13:27:12.241 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T13:27:12.242 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T13:27:12.297 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0cf46de4-f58b-4e68-a8b9-534bb33f8d72... 2024-09-16T13:27:12.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T13:27:12.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T13:27:12.379 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-09-16T13:27:12.379 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2024-09-16T13:27:12.379 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T13:27:12.380 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2024-09-16T13:27:12.381 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2024-09-16T13:27:12.382 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = "172.21.15.76,[v2:172.21.15.76:3301,v1:172.21.15.76:6790],172.21.15.163" 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on pool no app = False 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 5000 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 0cf46de4-f58b-4e68-a8b9-534bb33f8d72 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 600 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.383 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2024-09-16T13:27:12.384 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore compression algorithm = zlib 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore compression mode = aggressive 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 1/20 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 1/20 2024-09-16T13:27:12.385 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 4/10 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: osd max markdown count = 1000 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: osd shutdown pgref assert = True 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T13:27:12.386 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2024-09-16T13:27:12.387 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.388 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T13:27:12.389 INFO:teuthology.orchestra.run.smithi076.stdout: rbd cache = True 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi076.stdout: rbd cache max dirty = 0 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.c] 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.b] 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2024-09-16T13:27:12.390 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T13:27:12.391 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2024-09-16T13:27:12.392 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon_allow_pool_size_one = true 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.76,[v2:172.21.15.76:3301,v1:172.21.15.76:6790],172.21.15.163" 2024-09-16T13:27:12.393 INFO:teuthology.orchestra.run.smithi163.stdout: mon client directed command retry = 5 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on pool no app = False 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: ms inject socket failures = 5000 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 0cf46de4-f58b-4e68-a8b9-534bb33f8d72 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock profile = high_recovery_ops 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.394 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.395 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore block size = 96636764160 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore compression algorithm = zlib 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore compression mode = aggressive 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore fsck on mount = True 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluefs = 1/20 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluestore = 1/20 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: debug rocksdb = 4/10 2024-09-16T13:27:12.396 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd full ratio = 0.9 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd nearfull ratio = 0.8 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: osd failsafe full ratio = 0.95 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: osd max markdown count = 1000 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = bluestore 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: osd shutdown pgref assert = True 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:27:12.397 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:27:12.398 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: # 1m isn't quite enough 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: mon_down_mkfs_grace = 2m 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_filestore_osds = false 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2024-09-16T13:27:12.399 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: rbd cache = True 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout: rbd cache max dirty = 0 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2024-09-16T13:27:12.400 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2024-09-16T13:27:12.401 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T13:27:12.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T13:27:12.490 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T13:27:12.490 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:12.490 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T13:27:12.506 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:12.506 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T13:27:12.560 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:27:12.561 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:12.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T13:27:12.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:27:12.635 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:12.635 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:12.690 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:27:12.690 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:12.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T13:27:12.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:27:12.727 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:12.727 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:12.783 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T13:27:12.783 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T13:27:12.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T13:27:12.829 INFO:teuthology.orchestra.run.smithi076.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T13:27:12.832 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T13:27:12.888 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T13:27:12.891 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T13:27:12.891 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T13:27:12.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T13:27:12.928 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T13:27:12.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T13:27:12.978 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T13:27:12.991 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T13:27:12.991 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T13:27:12.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T13:27:13.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:13.015 INFO:tasks.ceph:role: osd.0 2024-09-16T13:27:13.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:27:13.015 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T13:27:13.087 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:13.087 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:13.087 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:13.087 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:13.087 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:13.088 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:13.088 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:13.088 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:13.088 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:13.088 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:13.580 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T13:27:13.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T13:27:13.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T13:27:13.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T13:27:13.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T13:27:13.778 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:13.778 INFO:tasks.ceph:role: osd.1 2024-09-16T13:27:13.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:27:13.778 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T13:27:13.849 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:13.850 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:14.311 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T13:27:14.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T13:27:14.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T13:27:14.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T13:27:14.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T13:27:14.521 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:14.521 INFO:tasks.ceph:role: osd.2 2024-09-16T13:27:14.521 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:27:14.522 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:14.593 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:14.594 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:14.594 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:15.072 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T13:27:15.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T13:27:15.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T13:27:15.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T13:27:15.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T13:27:15.260 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:15.260 INFO:tasks.ceph:role: osd.3 2024-09-16T13:27:15.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:27:15.261 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:15.335 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-09-16T13:27:15.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T13:27:15.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T13:27:15.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T13:27:15.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:16.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.089+0000 7fc86ad29440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T13:27:16.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.094+0000 7fc86ad29440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T13:27:16.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.094+0000 7fc86ad29440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:27:16.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.094+0000 7fc86ad29440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:27:16.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.094+0000 7fc86ad29440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:27:16.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.101+0000 7fc86ad29440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T13:27:16.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:16.629+0000 7fc86ad29440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:17.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:17.394+0000 7fc86ad29440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:18.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:18.158+0000 7fc86ad29440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:18.956 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T13:27:18.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:19.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.071+0000 7f5de9522440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T13:27:19.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.071+0000 7f5de9522440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T13:27:19.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.071+0000 7f5de9522440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:27:19.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.071+0000 7f5de9522440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:27:19.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.071+0000 7f5de9522440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:27:19.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.072+0000 7f5de9522440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T13:27:19.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:19.596+0000 7f5de9522440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:20.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:20.358+0000 7f5de9522440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:21.121+0000 7f5de9522440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:21.917 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T13:27:21.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:22.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.029+0000 7ff46cb1d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T13:27:22.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.030+0000 7ff46cb1d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T13:27:22.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.030+0000 7ff46cb1d440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:27:22.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.030+0000 7ff46cb1d440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:27:22.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.030+0000 7ff46cb1d440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:27:22.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.030+0000 7ff46cb1d440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T13:27:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:22.553+0000 7ff46cb1d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:23.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:23.319+0000 7ff46cb1d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.082+0000 7ff46cb1d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:24.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T13:27:24.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:24.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.997+0000 7f47ca388440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T13:27:24.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.997+0000 7f47ca388440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T13:27:24.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.997+0000 7f47ca388440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:27:24.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.997+0000 7f47ca388440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:27:24.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.997+0000 7f47ca388440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:27:25.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:24.998+0000 7f47ca388440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T13:27:25.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:25.529+0000 7f47ca388440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:26.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:26.297+0000 7f47ca388440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:27.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:27:27.059+0000 7f47ca388440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:27.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T13:27:27.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T13:27:27.924 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:27.924 INFO:tasks.ceph:role: osd.4 2024-09-16T13:27:27.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:27:27.924 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T13:27:28.001 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:28.001 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:28.001 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:28.504 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T13:27:28.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T13:27:28.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T13:27:28.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T13:27:28.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T13:27:28.746 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:28.746 INFO:tasks.ceph:role: osd.5 2024-09-16T13:27:28.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:27:28.746 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:28.822 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:28.822 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:28.822 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:29.272 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T13:27:29.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T13:27:29.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T13:27:29.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T13:27:29.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T13:27:29.460 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:29.460 INFO:tasks.ceph:role: osd.6 2024-09-16T13:27:29.460 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:27:29.460 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:29.533 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:29.534 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:29.534 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:29.534 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:29.534 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:30.013 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T13:27:30.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T13:27:30.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T13:27:30.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T13:27:30.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T13:27:30.262 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:27:30.262 INFO:tasks.ceph:role: osd.7 2024-09-16T13:27:30.262 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:27:30.263 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:27:30.335 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:27:30.336 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:27:30.336 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T13:27:30.336 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:27:30.336 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:27:30.858 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T13:27:30.860 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T13:27:30.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T13:27:31.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T13:27:31.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.138+0000 7f7e8c40d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.139+0000 7f7e8c40d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.139+0000 7f7e8c40d440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.139+0000 7f7e8c40d440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.139+0000 7f7e8c40d440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:27:31.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.139+0000 7f7e8c40d440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T13:27:31.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:31.664+0000 7f7e8c40d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:32.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:32.433+0000 7f7e8c40d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:33.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:33.200+0000 7f7e8c40d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:33.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T13:27:34.029 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.112+0000 7f98e481d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T13:27:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.113+0000 7f98e481d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T13:27:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.113+0000 7f98e481d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:27:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.113+0000 7f98e481d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:27:34.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.113+0000 7f98e481d440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:27:34.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.113+0000 7f98e481d440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T13:27:34.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:34.638+0000 7f98e481d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:35.400+0000 7f98e481d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:36.165+0000 7f98e481d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:36.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T13:27:36.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T13:27:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T13:27:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:27:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:27:37.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:27:37.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.080+0000 7ff86671d440 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T13:27:37.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:37.609+0000 7ff86671d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:38.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:38.372+0000 7ff86671d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:39.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:39.138+0000 7ff86671d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:39.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T13:27:39.966 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:40.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.050+0000 7fa653f1f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T13:27:40.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.050+0000 7fa653f1f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T13:27:40.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.050+0000 7fa653f1f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:27:40.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.050+0000 7fa653f1f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:27:40.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.050+0000 7fa653f1f440 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:27:40.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.051+0000 7fa653f1f440 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T13:27:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:40.581+0000 7fa653f1f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:41.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:41.345+0000 7fa653f1f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:42.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T13:27:42.109+0000 7fa653f1f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:27:42.909 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T13:27:42.939 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T13:27:42.939 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:42.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T13:27:42.970 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:42.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T13:27:43.039 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:43.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T13:27:43.109 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:43.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T13:27:43.177 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:43.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T13:27:43.243 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T13:27:43.274 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T13:27:43.343 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T13:27:43.411 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T13:27:43.481 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T13:27:43.551 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:43.551 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T13:27:43.569 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:43.569 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T13:27:43.609 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T13:27:43.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T13:27:43.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.y] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAwMuhmJURoMRAATU3bjom14ZT/IBEZqp7kYA== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA0MuhmyJlkBRAAhSQZTUJbL09xsPylqa4Vtw== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA3Muhms0lMBBAAavxctLG9CN9ncJUI+nf57Q== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA6MuhmGC/TARAAexwtufF0xzONCk1cyJX4nw== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.3] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA8Muhm3CB6OxAAkpuNSVLFWl5Um+BEVRyLag== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAwMuhmZWTtNBAAEuzgKvqHVXKBOSopUTtdqw== 2024-09-16T13:27:43.652 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.4] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBDMuhmUpVSCBAALXDXDtp4ln0HJGOvJ7aZJQ== 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.5] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBGMuhms5XGBhAA7026kBF4/OoB9Y8odpcZWw== 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.6] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBJMuhmtw7UBBAAEOKxKA5S1Z2v7fk6z3WMmQ== 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.7] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBMMuhm6O8MAxAAYZkMYrVog0S7I9BImIsbVA== 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAwMuhmaWNNNxAAFU7BO4vaOjhY6XqPTczUew== 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:[client.1] 2024-09-16T13:27:43.653 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAwMuhmDZFGOhAARtlhSA32peFYFI07TOxYzA== 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.y] 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQAwMuhmJURoMRAATU3bjom14ZT/IBEZqp7kYA== 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.0] 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA0MuhmyJlkBRAAhSQZTUJbL09xsPylqa4Vtw== 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.1] 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA3Muhms0lMBBAAavxctLG9CN9ncJUI+nf57Q== 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.2] 2024-09-16T13:27:43.675 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA6MuhmGC/TARAAexwtufF0xzONCk1cyJX4nw== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.3] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA8Muhm3CB6OxAAkpuNSVLFWl5Um+BEVRyLag== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.x] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQAwMuhmZWTtNBAAEuzgKvqHVXKBOSopUTtdqw== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.4] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBDMuhmUpVSCBAALXDXDtp4ln0HJGOvJ7aZJQ== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.5] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBGMuhms5XGBhAA7026kBF4/OoB9Y8odpcZWw== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.6] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBJMuhmtw7UBBAAEOKxKA5S1Z2v7fk6z3WMmQ== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.7] 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBMMuhm6O8MAxAAYZkMYrVog0S7I9BImIsbVA== 2024-09-16T13:27:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-09-16T13:27:43.677 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQAwMuhmaWNNNxAAFU7BO4vaOjhY6XqPTczUew== 2024-09-16T13:27:43.677 INFO:teuthology.orchestra.run.smithi163.stdout:[client.1] 2024-09-16T13:27:43.677 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQAwMuhmDZFGOhAARtlhSA32peFYFI07TOxYzA== 2024-09-16T13:27:43.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:27:43.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:27:43.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:43.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:27:43.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:27:44.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.013 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.059 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:27:44.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:27:44.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:27:44.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:27:44.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:27:44.306 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T13:27:44.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T13:27:44.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:27:44.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T13:27:44.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T13:27:44.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:27:44.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T13:27:44.625 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T13:27:44.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:27:44.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T13:27:44.767 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:44.769 DEBUG:teuthology.orchestra.run.smithi163:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:27:44.822 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T13:27:44.822 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T13:27:44.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T13:27:44.824 INFO:tasks.ceph.mon.a:Started 2024-09-16T13:27:44.825 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T13:27:44.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T13:27:44.826 INFO:tasks.ceph.mon.c:Started 2024-09-16T13:27:44.827 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T13:27:44.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T13:27:44.864 INFO:tasks.ceph.mon.b:Started 2024-09-16T13:27:44.864 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T13:27:44.864 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T13:27:44.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T13:27:44.867 INFO:tasks.ceph.mgr.y:Started 2024-09-16T13:27:44.867 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T13:27:44.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T13:27:44.869 INFO:tasks.ceph.mgr.x:Started 2024-09-16T13:27:44.869 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T13:27:44.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T13:27:50.246 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.244+0000 7fc5ae80fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T13:27:50.249 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.246+0000 7f90ce4ddec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T13:27:50.286 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.284+0000 7fc5ae80fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T13:27:50.289 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.286+0000 7f90ce4ddec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T13:27:50.335 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.333+0000 7fc5ae80fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T13:27:50.338 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.335+0000 7f90ce4ddec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T13:27:50.476 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.474+0000 7fc5ae80fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:27:50.486 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.483+0000 7f90ce4ddec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:27:50.582 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.580+0000 7fc5ae80fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T13:27:50.592 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.589+0000 7f90ce4ddec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T13:27:50.862 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.860+0000 7fc5ae80fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T13:27:50.870 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.867+0000 7f90ce4ddec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T13:27:50.917 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:50.915+0000 7fc5ae80fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T13:27:50.945 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:50.942+0000 7f90ce4ddec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T13:27:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-09-16T13:27:51.046 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T13:27:51.046 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:51.046 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T13:27:51.052 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:51.050+0000 7fc5ae80fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T13:27:51.071 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:51.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T13:27:51.076 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:51.073+0000 7f90ce4ddec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T13:27:51.137 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:51.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T13:27:51.201 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T13:27:51.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T13:27:51.266 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:51.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T13:27:51.291 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:51.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T13:27:51.356 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:51.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T13:27:51.421 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T13:27:51.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T13:27:51.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new daa6f01d-c106-4b1d-8063-1852db16dec3 0 2024-09-16T13:27:51.701 INFO:teuthology.orchestra.run.smithi163.stdout:0 2024-09-16T13:27:51.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 70bfb231-8371-4267-83d7-8c7bdf4d122b 1 2024-09-16T13:27:51.886 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-09-16T13:27:51.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new a4447a8c-bda5-4387-bc24-77c6cc28374b 2 2024-09-16T13:27:52.002 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.000+0000 7fc5ae80fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T13:27:52.007 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.004+0000 7f90ce4ddec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T13:27:52.051 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.049+0000 7fc5ae80fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T13:27:52.058 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.056+0000 7f90ce4ddec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T13:27:52.072 INFO:teuthology.orchestra.run.smithi163.stdout:2 2024-09-16T13:27:52.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 50031c12-2464-4373-b82a-2d4da1dae766 3 2024-09-16T13:27:52.258 INFO:teuthology.orchestra.run.smithi163.stdout:3 2024-09-16T13:27:52.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 24b2d993-ea4d-406e-adbd-e06d278de26d 4 2024-09-16T13:27:52.419 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.417+0000 7fc5ae80fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T13:27:52.439 INFO:teuthology.orchestra.run.smithi163.stdout:4 2024-09-16T13:27:52.442 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.440+0000 7f90ce4ddec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T13:27:52.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new d68602fa-7b2b-43fc-96ba-f807ca71872e 5 2024-09-16T13:27:52.501 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.499+0000 7fc5ae80fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T13:27:52.529 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.527+0000 7f90ce4ddec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T13:27:52.589 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.587+0000 7fc5ae80fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:27:52.621 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.619+0000 7f90ce4ddec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:27:52.623 INFO:teuthology.orchestra.run.smithi163.stdout:5 2024-09-16T13:27:52.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 5f02e46d-718f-454b-99c5-245ded452730 6 2024-09-16T13:27:52.635 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.633+0000 7fc5ae80fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T13:27:52.669 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.667+0000 7f90ce4ddec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T13:27:52.729 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.727+0000 7fc5ae80fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T13:27:52.767 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.765+0000 7f90ce4ddec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T13:27:52.781 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.780+0000 7fc5ae80fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T13:27:52.810 INFO:teuthology.orchestra.run.smithi163.stdout:6 2024-09-16T13:27:52.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new ea09d724-4028-4cd4-b5b5-9d0122dd5495 7 2024-09-16T13:27:52.822 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.820+0000 7f90ce4ddec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T13:27:52.826 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.823+0000 7fc5ae80fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T13:27:52.867 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.866+0000 7f90ce4ddec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T13:27:52.917 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.915+0000 7fc5ae80fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T13:27:52.964 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:52.962+0000 7f90ce4ddec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T13:27:52.966 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:52.964+0000 7fc5ae80fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T13:27:52.996 INFO:teuthology.orchestra.run.smithi163.stdout:7 2024-09-16T13:27:53.007 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T13:27:53.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T13:27:53.010 INFO:tasks.ceph.osd.0:Started 2024-09-16T13:27:53.010 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T13:27:53.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T13:27:53.012 INFO:tasks.ceph.osd.1:Started 2024-09-16T13:27:53.012 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T13:27:53.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T13:27:53.014 INFO:tasks.ceph.osd.2:Started 2024-09-16T13:27:53.014 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T13:27:53.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T13:27:53.015 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.012+0000 7f90ce4ddec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T13:27:53.017 INFO:tasks.ceph.osd.3:Started 2024-09-16T13:27:53.017 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T13:27:53.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T13:27:53.018 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:53.016+0000 7fc5ae80fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T13:27:53.019 INFO:tasks.ceph.osd.4:Started 2024-09-16T13:27:53.019 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T13:27:53.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T13:27:53.021 INFO:tasks.ceph.osd.5:Started 2024-09-16T13:27:53.021 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T13:27:53.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T13:27:53.024 INFO:tasks.ceph.osd.6:Started 2024-09-16T13:27:53.024 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T13:27:53.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T13:27:53.026 INFO:tasks.ceph.osd.7:Started 2024-09-16T13:27:53.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:27:53.089 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.087+0000 7f90ce4ddec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T13:27:53.120 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:53.119+0000 7fc5ae80fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T13:27:53.162 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.160+0000 7f90ce4ddec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T13:27:53.167 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:53.165+0000 7fc5ae80fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T13:27:53.204 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.202+0000 7f90ce4ddec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T13:27:53.205 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:27:53.205 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":10,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:27:52.988177+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:27:53.211 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:53.209+0000 7fc5ae80fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T13:27:53.216 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T13:27:53.217 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T13:27:53.245 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.244+0000 7f90ce4ddec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T13:27:53.608 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:27:53.606+0000 7f1ea2d2b440 -1 Falling back to public interface 2024-09-16T13:27:53.628 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:27:53.626+0000 7fca53412440 -1 Falling back to public interface 2024-09-16T13:27:53.630 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:27:53.628+0000 7f0690850440 -1 Falling back to public interface 2024-09-16T13:27:53.630 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:27:53.628+0000 7fd6a991d440 -1 Falling back to public interface 2024-09-16T13:27:53.631 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:27:53.629+0000 7fd6a7bfd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6811/0: (98) Address already in use 2024-09-16T13:27:53.631 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:27:53.629+0000 7f218fe8d440 -1 Falling back to public interface 2024-09-16T13:27:53.636 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:27:53.634+0000 7fd885610440 -1 Falling back to public interface 2024-09-16T13:27:53.640 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:27:53.638+0000 7f0d54c8b440 -1 Falling back to public interface 2024-09-16T13:27:53.648 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:27:53.646+0000 7f3f8403b440 -1 Falling back to public interface 2024-09-16T13:27:53.831 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:53.829+0000 7fc5ae80fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T13:27:53.860 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T13:27:53.858+0000 7f90ce4ddec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T13:27:54.022 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:27:54.173 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T13:27:54.173 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":10,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:27:52.988177+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:27:54.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T13:27:54.842 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:54.840+0000 7fc593494640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T13:27:54.842 INFO:tasks.ceph.mgr.y.smithi076.stderr:2024-09-16T13:27:54.841+0000 7fc593494640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T13:27:57.049 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:27:57.047+0000 7f1ea2d2b440 -1 osd.1 0 log_to_monitors true 2024-09-16T13:27:57.089 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:27:57.087+0000 7fd885610440 -1 osd.2 0 log_to_monitors true 2024-09-16T13:27:57.097 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:27:57.095+0000 7fca53412440 -1 osd.6 0 log_to_monitors true 2024-09-16T13:27:57.129 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:27:57.127+0000 7f0d54c8b440 -1 osd.0 0 log_to_monitors true 2024-09-16T13:27:57.144 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:27:57.143+0000 7f3f8403b440 -1 osd.3 0 log_to_monitors true 2024-09-16T13:27:57.154 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:27:57.153+0000 7f0690850440 -1 osd.4 0 log_to_monitors true 2024-09-16T13:27:57.171 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:27:57.169+0000 7fd6a991d440 -1 osd.7 0 log_to_monitors true 2024-09-16T13:27:57.172 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:27:57.170+0000 7f218fe8d440 -1 osd.5 0 log_to_monitors true 2024-09-16T13:28:00.990 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:28:01.210 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:01.208+0000 7fca4f3f9640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T13:28:01.213 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:01.212+0000 7fca46be8640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:01.372 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:01.370+0000 7f1e9eff9640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T13:28:01.380 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:01.377+0000 7f1e967e8640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:01.441 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:01.439+0000 7fd87fdf6640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T13:28:01.443 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T13:28:01.443 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":13,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:27:59.143044+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:28:01.447 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:01.445+0000 7fd878de8640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:01.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T13:28:01.466 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:01.464+0000 7f0d4fdf6640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T13:28:01.469 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:01.468+0000 7f0d48de8640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:02.263 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:02.261+0000 7f3f7f1f6640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T13:28:02.267 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:02.265+0000 7f3f78084640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:02.434 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:02.432+0000 7f218aff6640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T13:28:02.440 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:02.438+0000 7f2183fe8640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:02.515 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:02.513+0000 7f068c7f9640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T13:28:02.521 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:02.519+0000 7f0683fe8640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:02.526 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:02.525+0000 7fd6a43f6640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T13:28:02.530 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:02.528+0000 7fd69d3e8640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:28:08.262 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:28:08.532 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T13:28:08.532 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":18,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:28:06.285502+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:28:08.543 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T13:28:08.543 INFO:tasks.ceph:Creating RBD pool 2024-09-16T13:28:08.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T13:28:09.302 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'rbd' created 2024-09-16T13:28:09.316 DEBUG:teuthology.orchestra.run.smithi076:> rbd --cluster ceph pool init rbd 2024-09-16T13:28:12.324 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T13:28:12.325 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T13:28:12.325 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T13:28:12.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T13:28:12.326 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T13:28:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:12.651 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":33145},{"type":"v1","addr":"172.21.15.76:6833","nonce":33145}]},"active_addr":"172.21.15.76:6833/33145","active_change":"2024-09-16T13:27:53.831566+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2204424505}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":1399189697}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":992953133}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":1610296970}]}]} 2024-09-16T13:28:12.655 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T13:28:12.655 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T13:28:12.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:28:12.918 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:12.918 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:28:12.301583+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:28:12.930 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T13:28:12.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:28:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:13.194 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:28:12.301583+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:27:52.988177+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:28:13.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T13:28:13.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T13:28:13.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T13:28:13.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T13:28:13.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T13:28:13.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T13:28:13.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T13:28:13.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T13:28:13.450 INFO:teuthology.orchestra.run.smithi076.stdout:60129542152 2024-09-16T13:28:13.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:28:13.452 INFO:teuthology.orchestra.run.smithi076.stdout:60129542152 2024-09-16T13:28:13.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:28:13.453 INFO:teuthology.orchestra.run.smithi076.stdout:64424509447 2024-09-16T13:28:13.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:28:13.461 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:13.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:28:13.470 INFO:teuthology.orchestra.run.smithi076.stdout:64424509446 2024-09-16T13:28:13.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:28:13.473 INFO:teuthology.orchestra.run.smithi076.stdout:64424509445 2024-09-16T13:28:13.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:28:13.480 INFO:teuthology.orchestra.run.smithi076.stdout:64424509445 2024-09-16T13:28:13.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:28:13.491 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:13.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:28:14.023 INFO:teuthology.orchestra.run.smithi076.stdout:64424509446 2024-09-16T13:28:14.024 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:14.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.5 2024-09-16T13:28:14.047 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:14.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.6 2024-09-16T13:28:14.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-09-16T13:28:14.079 INFO:teuthology.orchestra.run.smithi076.stdout:64424509444 2024-09-16T13:28:14.086 INFO:teuthology.orchestra.run.smithi076.stdout:64424509444 2024-09-16T13:28:14.088 INFO:teuthology.orchestra.run.smithi076.stdout:64424509445 2024-09-16T13:28:14.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.7 2024-09-16T13:28:14.103 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.3 2024-09-16T13:28:14.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2024-09-16T13:28:14.126 INFO:teuthology.orchestra.run.smithi076.stdout:60129542148 2024-09-16T13:28:14.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.0 2024-09-16T13:28:14.187 INFO:teuthology.orchestra.run.smithi076.stdout:60129542148 2024-09-16T13:28:14.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.2 2024-09-16T13:28:15.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:28:15.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:28:15.059 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:28:15.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:28:15.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:28:15.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:28:15.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:28:15.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:28:15.497 INFO:teuthology.orchestra.run.smithi076.stdout:64424509447 2024-09-16T13:28:15.515 INFO:teuthology.orchestra.run.smithi076.stdout:60129542152 2024-09-16T13:28:15.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2024-09-16T13:28:15.516 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-09-16T13:28:15.534 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.593 INFO:teuthology.orchestra.run.smithi076.stdout:64424509445 2024-09-16T13:28:15.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2024-09-16T13:28:15.605 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.618 INFO:teuthology.orchestra.run.smithi076.stdout:60129542152 2024-09-16T13:28:15.626 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:15.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-09-16T13:28:15.633 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2024-09-16T13:28:15.639 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.646 INFO:teuthology.orchestra.run.smithi076.stdout:64424509446 2024-09-16T13:28:15.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.4 2024-09-16T13:28:15.658 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.660 INFO:teuthology.orchestra.run.smithi076.stdout:64424509445 2024-09-16T13:28:15.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.7 2024-09-16T13:28:15.677 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.702 INFO:teuthology.orchestra.run.smithi076.stdout:60129542150 2024-09-16T13:28:15.713 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.0 2024-09-16T13:28:15.714 DEBUG:teuthology.parallel:result is None 2024-09-16T13:28:15.714 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:28:15.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:16.019 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:16.020 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:16.031 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T13:28:15.839557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2380712,"kb_used_data":1848,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752594008,"statfs":{"total":773094113280,"available":770656264192,"internally_reserved":0,"allocated":1892352,"data_stored":1782502,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.538634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.801240+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:12.801240+0000","last_peered":"2024-09-16T13:28:12.801240+0000","last_clean":"2024-09-16T13:28:12.801240+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:12.801240+0000","last_undegraded":"2024-09-16T13:28:12.801240+0000","last_fullsized":"2024-09-16T13:28:12.801240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.311930+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:12.311930+0000","last_peered":"2024-09-16T13:28:12.311930+0000","last_clean":"2024-09-16T13:28:12.311930+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:12.311930+0000","last_undegraded":"2024-09-16T13:28:12.311930+0000","last_fullsized":"2024-09-16T13:28:12.311930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.959056+0000","last_change":"2024-09-16T13:28:12.959294+0000","last_active":"2024-09-16T13:28:12.959056+0000","last_peered":"2024-09-16T13:28:12.959056+0000","last_clean":"2024-09-16T13:28:12.959056+0000","last_became_active":"2024-09-16T13:28:10.306929+0000","last_became_peered":"2024-09-16T13:28:10.306929+0000","last_unstale":"2024-09-16T13:28:12.959056+0000","last_undegraded":"2024-09-16T13:28:12.959056+0000","last_fullsized":"2024-09-16T13:28:12.959056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:42.738385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034515700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.311880+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:12.311880+0000","last_peered":"2024-09-16T13:28:12.311880+0000","last_clean":"2024-09-16T13:28:12.311880+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:12.311880+0000","last_undegraded":"2024-09-16T13:28:12.311880+0000","last_fullsized":"2024-09-16T13:28:12.311880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.317911+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:12.317911+0000","last_peered":"2024-09-16T13:28:12.317911+0000","last_clean":"2024-09-16T13:28:12.317911+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:12.317911+0000","last_undegraded":"2024-09-16T13:28:12.317911+0000","last_fullsized":"2024-09-16T13:28:12.317911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.312998+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:12.312998+0000","last_peered":"2024-09-16T13:28:12.312998+0000","last_clean":"2024-09-16T13:28:12.312998+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:12.312998+0000","last_undegraded":"2024-09-16T13:28:12.312998+0000","last_fullsized":"2024-09-16T13:28:12.312998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.959003+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:12.959003+0000","last_peered":"2024-09-16T13:28:12.959003+0000","last_clean":"2024-09-16T13:28:12.959003+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:12.959003+0000","last_undegraded":"2024-09-16T13:28:12.959003+0000","last_fullsized":"2024-09-16T13:28:12.959003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.317142+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:12.317142+0000","last_peered":"2024-09-16T13:28:12.317142+0000","last_clean":"2024-09-16T13:28:12.317142+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:12.317142+0000","last_undegraded":"2024-09-16T13:28:12.317142+0000","last_fullsized":"2024-09-16T13:28:12.317142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'72","reported_seq":102,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:28:10.304936+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:10.304936+0000","last_peered":"2024-09-16T13:28:10.304936+0000","last_clean":"2024-09-16T13:28:10.304936+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:10.304936+0000","last_undegraded":"2024-09-16T13:28:10.304936+0000","last_fullsized":"2024-09-16T13:28:10.304936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":665584,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":225280,"data_stored":75235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":225280,"data_stored":75235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":665584,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:16.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:16.290 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:16.290 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:16.302 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T13:28:15.839557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2380712,"kb_used_data":1848,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752594008,"statfs":{"total":773094113280,"available":770656264192,"internally_reserved":0,"allocated":1892352,"data_stored":1782502,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.538634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.801240+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:12.801240+0000","last_peered":"2024-09-16T13:28:12.801240+0000","last_clean":"2024-09-16T13:28:12.801240+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:12.801240+0000","last_undegraded":"2024-09-16T13:28:12.801240+0000","last_fullsized":"2024-09-16T13:28:12.801240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.311930+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:12.311930+0000","last_peered":"2024-09-16T13:28:12.311930+0000","last_clean":"2024-09-16T13:28:12.311930+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:12.311930+0000","last_undegraded":"2024-09-16T13:28:12.311930+0000","last_fullsized":"2024-09-16T13:28:12.311930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.959056+0000","last_change":"2024-09-16T13:28:12.959294+0000","last_active":"2024-09-16T13:28:12.959056+0000","last_peered":"2024-09-16T13:28:12.959056+0000","last_clean":"2024-09-16T13:28:12.959056+0000","last_became_active":"2024-09-16T13:28:10.306929+0000","last_became_peered":"2024-09-16T13:28:10.306929+0000","last_unstale":"2024-09-16T13:28:12.959056+0000","last_undegraded":"2024-09-16T13:28:12.959056+0000","last_fullsized":"2024-09-16T13:28:12.959056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:42.738385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034515700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.311880+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:12.311880+0000","last_peered":"2024-09-16T13:28:12.311880+0000","last_clean":"2024-09-16T13:28:12.311880+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:12.311880+0000","last_undegraded":"2024-09-16T13:28:12.311880+0000","last_fullsized":"2024-09-16T13:28:12.311880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.317911+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:12.317911+0000","last_peered":"2024-09-16T13:28:12.317911+0000","last_clean":"2024-09-16T13:28:12.317911+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:12.317911+0000","last_undegraded":"2024-09-16T13:28:12.317911+0000","last_fullsized":"2024-09-16T13:28:12.317911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.312998+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:12.312998+0000","last_peered":"2024-09-16T13:28:12.312998+0000","last_clean":"2024-09-16T13:28:12.312998+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:12.312998+0000","last_undegraded":"2024-09-16T13:28:12.312998+0000","last_fullsized":"2024-09-16T13:28:12.312998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.959003+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:12.959003+0000","last_peered":"2024-09-16T13:28:12.959003+0000","last_clean":"2024-09-16T13:28:12.959003+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:12.959003+0000","last_undegraded":"2024-09-16T13:28:12.959003+0000","last_fullsized":"2024-09-16T13:28:12.959003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"20'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T13:28:12.317142+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:12.317142+0000","last_peered":"2024-09-16T13:28:12.317142+0000","last_clean":"2024-09-16T13:28:12.317142+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:12.317142+0000","last_undegraded":"2024-09-16T13:28:12.317142+0000","last_fullsized":"2024-09-16T13:28:12.317142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"18'72","reported_seq":102,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:28:10.304936+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:10.304936+0000","last_peered":"2024-09-16T13:28:10.304936+0000","last_clean":"2024-09-16T13:28:10.304936+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:10.304936+0000","last_undegraded":"2024-09-16T13:28:10.304936+0000","last_fullsized":"2024-09-16T13:28:10.304936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":665584,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":225280,"data_stored":75235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":221184,"data_stored":75216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":225280,"data_stored":75235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":278528,"data_stored":665584,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:16.303 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:28:16.303 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T13:28:16.304 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T13:28:16.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T13:28:16.604 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T13:28:16.604 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T13:28:16.615 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T13:28:16.615 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:28:16.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-09-16T13:28:16.919 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cache' created 2024-09-16T13:28:16.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2024-09-16T13:28:17.925 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2024-09-16T13:28:17.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-09-16T13:28:18.929 INFO:teuthology.orchestra.run.smithi076.stderr:set cache-mode for pool 'cache' to writeback 2024-09-16T13:28:18.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2024-09-16T13:28:19.933 INFO:teuthology.orchestra.run.smithi076.stderr:overlay for 'rbd' is now (or already was) 'cache' 2024-09-16T13:28:19.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-09-16T13:28:20.936 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 3 hit_set_type to bloom 2024-09-16T13:28:20.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-09-16T13:28:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 3 hit_set_count to 8 2024-09-16T13:28:21.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-09-16T13:28:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 3 hit_set_period to 60 2024-09-16T13:28:22.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-09-16T13:28:23.964 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 3 target_max_objects to 250 2024-09-16T13:28:23.978 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T13:28:23.988 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T13:28:23.988 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache max dirty': 0}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T13:28:23.989 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T13:28:23.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T13:28:24.273 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T13:28:24.273 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:28:24.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:24.535 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:24.546 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-16T13:28:23.958483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381212,"kb_used_data":2328,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752593508,"statfs":{"total":773094113280,"available":770655752192,"internally_reserved":0,"allocated":2383872,"data_stored":2026130,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.932229+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:20.932229+0000","last_peered":"2024-09-16T13:28:20.932229+0000","last_clean":"2024-09-16T13:28:20.932229+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:20.932229+0000","last_undegraded":"2024-09-16T13:28:20.932229+0000","last_fullsized":"2024-09-16T13:28:20.932229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934822+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934822+0000","last_peered":"2024-09-16T13:28:20.934822+0000","last_clean":"2024-09-16T13:28:20.934822+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:20.934822+0000","last_undegraded":"2024-09-16T13:28:20.934822+0000","last_fullsized":"2024-09-16T13:28:20.934822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916757+0000","last_change":"2024-09-16T13:28:12.959294+0000","last_active":"2024-09-16T13:28:16.916757+0000","last_peered":"2024-09-16T13:28:16.916757+0000","last_clean":"2024-09-16T13:28:16.916757+0000","last_became_active":"2024-09-16T13:28:10.306929+0000","last_became_peered":"2024-09-16T13:28:10.306929+0000","last_unstale":"2024-09-16T13:28:16.916757+0000","last_undegraded":"2024-09-16T13:28:16.916757+0000","last_fullsized":"2024-09-16T13:28:16.916757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:42.738385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034515700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934809+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934809+0000","last_peered":"2024-09-16T13:28:20.934809+0000","last_clean":"2024-09-16T13:28:20.934809+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:20.934809+0000","last_undegraded":"2024-09-16T13:28:20.934809+0000","last_fullsized":"2024-09-16T13:28:20.934809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.927666+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:17.927666+0000","last_peered":"2024-09-16T13:28:17.927666+0000","last_clean":"2024-09-16T13:28:17.927666+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:17.927666+0000","last_undegraded":"2024-09-16T13:28:17.927666+0000","last_fullsized":"2024-09-16T13:28:17.927666+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.325032+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:17.325032+0000","last_peered":"2024-09-16T13:28:17.325032+0000","last_clean":"2024-09-16T13:28:17.325032+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:17.325032+0000","last_undegraded":"2024-09-16T13:28:17.325032+0000","last_fullsized":"2024-09-16T13:28:17.325032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934859+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:28:20.934859+0000","last_peered":"2024-09-16T13:28:20.934859+0000","last_clean":"2024-09-16T13:28:20.934859+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:28:20.934859+0000","last_undegraded":"2024-09-16T13:28:20.934859+0000","last_fullsized":"2024-09-16T13:28:20.934859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.994776+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:21.994776+0000","last_peered":"2024-09-16T13:28:21.994776+0000","last_clean":"2024-09-16T13:28:21.994776+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:21.994776+0000","last_undegraded":"2024-09-16T13:28:21.994776+0000","last_fullsized":"2024-09-16T13:28:21.994776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.949880+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:28:21.949880+0000","last_peered":"2024-09-16T13:28:21.949880+0000","last_clean":"2024-09-16T13:28:21.949880+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:28:21.949880+0000","last_undegraded":"2024-09-16T13:28:21.949880+0000","last_fullsized":"2024-09-16T13:28:21.949880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:31.394402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916736+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:16.916736+0000","last_peered":"2024-09-16T13:28:16.916736+0000","last_clean":"2024-09-16T13:28:16.916736+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:16.916736+0000","last_undegraded":"2024-09-16T13:28:16.916736+0000","last_fullsized":"2024-09-16T13:28:16.916736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T13:28:18.024002+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:18.024002+0000","last_peered":"2024-09-16T13:28:18.024002+0000","last_clean":"2024-09-16T13:28:18.024002+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:18.024002+0000","last_undegraded":"2024-09-16T13:28:18.024002+0000","last_fullsized":"2024-09-16T13:28:18.024002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:52.853771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.325035+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:17.325035+0000","last_peered":"2024-09-16T13:28:17.325035+0000","last_clean":"2024-09-16T13:28:17.325035+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:17.325035+0000","last_undegraded":"2024-09-16T13:28:17.325035+0000","last_fullsized":"2024-09-16T13:28:17.325035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":105,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916747+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:16.916747+0000","last_peered":"2024-09-16T13:28:16.916747+0000","last_clean":"2024-09-16T13:28:16.916747+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:16.916747+0000","last_undegraded":"2024-09-16T13:28:16.916747+0000","last_fullsized":"2024-09-16T13:28:16.916747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":683621,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297620,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074220,"statfs":{"total":96636764160,"available":96332001280,"internally_reserved":0,"allocated":262144,"data_stored":93272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297620,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074220,"statfs":{"total":96636764160,"available":96332001280,"internally_reserved":0,"allocated":270336,"data_stored":99352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297612,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074228,"statfs":{"total":96636764160,"available":96332009472,"internally_reserved":0,"allocated":258048,"data_stored":93253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":294912,"data_stored":111948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":319488,"data_stored":124250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":299008,"data_stored":111967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":364544,"data_stored":708467,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:24.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:24.807 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:24.821 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-16T13:28:23.958483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381212,"kb_used_data":2328,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752593508,"statfs":{"total":773094113280,"available":770655752192,"internally_reserved":0,"allocated":2383872,"data_stored":2026130,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.932229+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:20.932229+0000","last_peered":"2024-09-16T13:28:20.932229+0000","last_clean":"2024-09-16T13:28:20.932229+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:20.932229+0000","last_undegraded":"2024-09-16T13:28:20.932229+0000","last_fullsized":"2024-09-16T13:28:20.932229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934822+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934822+0000","last_peered":"2024-09-16T13:28:20.934822+0000","last_clean":"2024-09-16T13:28:20.934822+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:20.934822+0000","last_undegraded":"2024-09-16T13:28:20.934822+0000","last_fullsized":"2024-09-16T13:28:20.934822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916757+0000","last_change":"2024-09-16T13:28:12.959294+0000","last_active":"2024-09-16T13:28:16.916757+0000","last_peered":"2024-09-16T13:28:16.916757+0000","last_clean":"2024-09-16T13:28:16.916757+0000","last_became_active":"2024-09-16T13:28:10.306929+0000","last_became_peered":"2024-09-16T13:28:10.306929+0000","last_unstale":"2024-09-16T13:28:16.916757+0000","last_undegraded":"2024-09-16T13:28:16.916757+0000","last_fullsized":"2024-09-16T13:28:16.916757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:42.738385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034515700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934809+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934809+0000","last_peered":"2024-09-16T13:28:20.934809+0000","last_clean":"2024-09-16T13:28:20.934809+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:20.934809+0000","last_undegraded":"2024-09-16T13:28:20.934809+0000","last_fullsized":"2024-09-16T13:28:20.934809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.927666+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:17.927666+0000","last_peered":"2024-09-16T13:28:17.927666+0000","last_clean":"2024-09-16T13:28:17.927666+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:17.927666+0000","last_undegraded":"2024-09-16T13:28:17.927666+0000","last_fullsized":"2024-09-16T13:28:17.927666+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.325032+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:17.325032+0000","last_peered":"2024-09-16T13:28:17.325032+0000","last_clean":"2024-09-16T13:28:17.325032+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:17.325032+0000","last_undegraded":"2024-09-16T13:28:17.325032+0000","last_fullsized":"2024-09-16T13:28:17.325032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934859+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:28:20.934859+0000","last_peered":"2024-09-16T13:28:20.934859+0000","last_clean":"2024-09-16T13:28:20.934859+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:28:20.934859+0000","last_undegraded":"2024-09-16T13:28:20.934859+0000","last_fullsized":"2024-09-16T13:28:20.934859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.994776+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:21.994776+0000","last_peered":"2024-09-16T13:28:21.994776+0000","last_clean":"2024-09-16T13:28:21.994776+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:21.994776+0000","last_undegraded":"2024-09-16T13:28:21.994776+0000","last_fullsized":"2024-09-16T13:28:21.994776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.949880+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:28:21.949880+0000","last_peered":"2024-09-16T13:28:21.949880+0000","last_clean":"2024-09-16T13:28:21.949880+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:28:21.949880+0000","last_undegraded":"2024-09-16T13:28:21.949880+0000","last_fullsized":"2024-09-16T13:28:21.949880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:31.394402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916736+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:16.916736+0000","last_peered":"2024-09-16T13:28:16.916736+0000","last_clean":"2024-09-16T13:28:16.916736+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:16.916736+0000","last_undegraded":"2024-09-16T13:28:16.916736+0000","last_fullsized":"2024-09-16T13:28:16.916736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T13:28:18.024002+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:18.024002+0000","last_peered":"2024-09-16T13:28:18.024002+0000","last_clean":"2024-09-16T13:28:18.024002+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:18.024002+0000","last_undegraded":"2024-09-16T13:28:18.024002+0000","last_fullsized":"2024-09-16T13:28:18.024002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:52.853771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:17.325035+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:17.325035+0000","last_peered":"2024-09-16T13:28:17.325035+0000","last_clean":"2024-09-16T13:28:17.325035+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:17.325035+0000","last_undegraded":"2024-09-16T13:28:17.325035+0000","last_fullsized":"2024-09-16T13:28:17.325035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":105,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-16T13:28:16.916747+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:16.916747+0000","last_peered":"2024-09-16T13:28:16.916747+0000","last_clean":"2024-09-16T13:28:16.916747+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:16.916747+0000","last_undegraded":"2024-09-16T13:28:16.916747+0000","last_fullsized":"2024-09-16T13:28:16.916747+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297668,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074172,"statfs":{"total":96636764160,"available":96331952128,"internally_reserved":0,"allocated":315392,"data_stored":683621,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297620,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074220,"statfs":{"total":96636764160,"available":96332001280,"internally_reserved":0,"allocated":262144,"data_stored":93272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297620,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074220,"statfs":{"total":96636764160,"available":96332001280,"internally_reserved":0,"allocated":270336,"data_stored":99352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297612,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074228,"statfs":{"total":96636764160,"available":96332009472,"internally_reserved":0,"allocated":258048,"data_stored":93253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":294912,"data_stored":111948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":319488,"data_stored":124250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":299008,"data_stored":111967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":364544,"data_stored":708467,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:24.822 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:28:24.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T13:28:25.090 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 31 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 0cf46de4-f58b-4e68-a8b9-534bb33f8d72 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-09-16T13:27:50.121006+0000 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-09-16T13:28:23.951903+0000 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 5 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client jewel 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release quincy 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2024-09-16T13:28:25.091 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 27 lfor 27/27/27 flags hashpspool,selfmanaged_snaps tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:pool 3 'cache' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 31 lfor 27/27/27 flags hashpspool,incomplete_clones,selfmanaged_snaps tier_of 2 cache_mode writeback target_objects 250 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 60s x8 decay_rate 0 search_last_n 0 stripe_width 0 application rbd 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 8 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6816/33372,v1:172.21.15.76:6817/33372] [v2:172.21.15.76:6818/33372,v1:172.21.15.76:6819/33372] exists,up daa6f01d-c106-4b1d-8063-1852db16dec3 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6800/33371,v1:172.21.15.76:6801/33371] [v2:172.21.15.76:6802/33371,v1:172.21.15.76:6803/33371] exists,up 70bfb231-8371-4267-83d7-8c7bdf4d122b 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6808/33387,v1:172.21.15.76:6809/33387] [v2:172.21.15.76:6810/33387,v1:172.21.15.76:6811/33387] exists,up a4447a8c-bda5-4387-bc24-77c6cc28374b 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6824/33388,v1:172.21.15.76:6825/33388] [v2:172.21.15.76:6826/33388,v1:172.21.15.76:6827/33388] exists,up 50031c12-2464-4373-b82a-2d4da1dae766 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6808/33329,v1:172.21.15.163:6809/33329] [v2:172.21.15.163:6811/33329,v1:172.21.15.163:6813/33329] exists,up 24b2d993-ea4d-406e-adbd-e06d278de26d 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6817/33357,v1:172.21.15.163:6820/33357] [v2:172.21.15.163:6823/33357,v1:172.21.15.163:6827/33357] exists,up d68602fa-7b2b-43fc-96ba-f807ca71872e 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6800/33330,v1:172.21.15.163:6801/33330] [v2:172.21.15.163:6802/33330,v1:172.21.15.163:6803/33330] exists,up 5f02e46d-718f-454b-99c5-245ded452730 2024-09-16T13:28:25.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6810/33331,v1:172.21.15.163:6812/33331] [v2:172.21.15.163:6814/33331,v1:172.21.15.163:6816/33331] exists,up ea09d724-4028-4cd4-b5b5-9d0122dd5495 2024-09-16T13:28:25.103 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6816/33372,v1:172.21.15.76:6817/33372] [v2:172.21.15.76:6818/33372,v1:172.21.15.76:6819/33372] exists,up daa6f01d-c106-4b1d-8063-1852db16dec3', 'osd.1 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6800/33371,v1:172.21.15.76:6801/33371] [v2:172.21.15.76:6802/33371,v1:172.21.15.76:6803/33371] exists,up 70bfb231-8371-4267-83d7-8c7bdf4d122b', 'osd.2 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6808/33387,v1:172.21.15.76:6809/33387] [v2:172.21.15.76:6810/33387,v1:172.21.15.76:6811/33387] exists,up a4447a8c-bda5-4387-bc24-77c6cc28374b', 'osd.3 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6824/33388,v1:172.21.15.76:6825/33388] [v2:172.21.15.76:6826/33388,v1:172.21.15.76:6827/33388] exists,up 50031c12-2464-4373-b82a-2d4da1dae766', 'osd.4 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6808/33329,v1:172.21.15.163:6809/33329] [v2:172.21.15.163:6811/33329,v1:172.21.15.163:6813/33329] exists,up 24b2d993-ea4d-406e-adbd-e06d278de26d', 'osd.5 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6817/33357,v1:172.21.15.163:6820/33357] [v2:172.21.15.163:6823/33357,v1:172.21.15.163:6827/33357] exists,up d68602fa-7b2b-43fc-96ba-f807ca71872e', 'osd.6 up in weight 1 up_from 14 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6800/33330,v1:172.21.15.163:6801/33330] [v2:172.21.15.163:6802/33330,v1:172.21.15.163:6803/33330] exists,up 5f02e46d-718f-454b-99c5-245ded452730', 'osd.7 up in weight 1 up_from 15 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6810/33331,v1:172.21.15.163:6812/33331] [v2:172.21.15.163:6814/33331,v1:172.21.15.163:6816/33331] exists,up ea09d724-4028-4cd4-b5b5-9d0122dd5495'] 2024-09-16T13:28:25.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T13:28:25.197 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:25.197 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mon.a", 2024-09-16T13:28:25.197 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster": "ceph", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket_mode": "", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_client_required": "cephx, none", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_cluster_required": "cephx", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_debug": "true", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_required": "cephx", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_supported": "cephx", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio": "true", 2024-09-16T13:28:25.198 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_reap_max": "16", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_async_discard": "false", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_block_size": "4096", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio": "false", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_enable_discard": "false", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry": "3", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash": "0", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring": "false", 2024-09-16T13:28:25.199 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_hipri": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_type": "", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_allocator": "hybrid", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_buffered_io": "true", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T13:28:25.200 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery": "false", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_sync_write": "false", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T13:28:25.201 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_allocator": "hybrid", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_async_db_compaction": "true", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T13:28:25.202 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_create": "true", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_create": "false", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_path": "", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_size": "0", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_path": "", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_size": "107374182400", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_create": "false", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_path": "", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T13:28:25.203 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs": "true", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune": "true", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T13:28:25.204 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size": "0", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_type": "2q", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_clone_cow": "true", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T13:28:25.205 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_mode": "none", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_freelist": "false", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T13:28:25.206 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T13:28:25.207 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fail_eio": "false", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T13:28:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T13:28:25.209 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size": "0", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T13:28:25.210 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_mem": "512", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T13:28:25.211 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_tracing": "false", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T13:28:25.212 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T13:28:25.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_version": "2", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_signatures": "false", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_version": "2", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_signatures": "false", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_version": "2", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_sign_messages": "true", 2024-09-16T13:28:25.214 INFO:teuthology.orchestra.run.smithi076.stdout: "chdir": "", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_acl_type": "", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_asio_thread_count": "2", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_mid": "0.750000", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_size": "16384", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_caps_release_delay": "5", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_check_pool_perm": "true", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_force_sync_read": "false", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_getattr_caps": "false", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_inject_features": "", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_remount": "false", 2024-09-16T13:28:25.215 INFO:teuthology.orchestra.run.smithi076.stdout: "client_dirsize_rbytes": "true", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_force_lazyio": "false", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_fs": "", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_release_failure": "false", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_max_inline_size": "4096", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mds_namespace": "", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_metadata": "", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_gid": "-1", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_timeout": "300", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_uid": "-1", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mountpoint": "/", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_notify_timeout": "10", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc": "true", 2024-09-16T13:28:25.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_objects": "1000", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_size": "209715200", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_permissions": "true", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_quota": "true", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_quota_df": "true", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_bytes": "0", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_periods": "4", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_min": "131072", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_reconnect_stale": "false", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_shutdown_timeout": "30", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_snapdir": ".snap", 2024-09-16T13:28:25.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_tick_interval": "1", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "client_trace": "", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_faked_inos": "false", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_random_mds": "false", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog": "false", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_port": "12201", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_monitors": "default=true", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog": "false", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_level": "info", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "-", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network": "", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network_interface": "", 2024-09-16T13:28:25.218 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_isal": "false", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_level": "5", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zstd_level": "1", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook": "", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook_timeout": "10", 2024-09-16T13:28:25.219 INFO:teuthology.orchestra.run.smithi076.stdout: "daemonize": "false", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_alienstore": "0/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok": "1/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok_assert_abort": "false", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_auth": "20/20", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bdev": "1/3", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluefs": "1/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluestore": "1/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_buffer": "0/1", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cephsqlite": "0/5", 2024-09-16T13:28:25.220 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_civetweb": "1/10", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_client": "0/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_compressor": "1/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_context": "0/1", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crush": "1/1", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crypto": "1/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_dpdk": "1/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_eventtrace": "1/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filer": "0/1", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filestore": "1/3", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_finisher": "1/1", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_fuse": "1/5", 2024-09-16T13:28:25.221 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_javaclient": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journal": "1/3", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journaler": "0/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_kstore": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_leveldb": "4/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_lockdep": "0/1", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mclock": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_balancer": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_locker": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_migrator": "1/5", 2024-09-16T13:28:25.222 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_memdb": "4/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgr": "2/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgrc": "1/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mon": "20/20", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_monc": "0/10", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_ms": "1/1", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_none": "0/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objclass": "0/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objectcacher": "0/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objecter": "0/1", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_optracker": "0/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_osd": "1/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_paxos": "20/20", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_perfcounter": "1/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_prioritycache": "1/5", 2024-09-16T13:28:25.223 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rados": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_replay": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_refs": "0/0", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_reserver": "1/1", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw": "1/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_sync": "1/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rocksdb": "4/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_cache": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_device": "0/5", 2024-09-16T13:28:25.224 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_journal": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_lba": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_odata": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_omap": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_onode": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_seastore_tm": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_striper": "0/1", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_test": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_throttle": "1/1", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_timer": "0/1", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_tp": "0/5", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "device_failure_prediction_mode": "none", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T13:28:25.225 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_graylog": "false", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_journald": "false", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_stderr": "true", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_syslog": "false", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event_tracing": "false", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_http_port": "9926", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_prio_limit": "5", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_sort_metrics": "true", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "exporter_stats_period": "5", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "fake_statfs_for_testing": "0", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "fatal_signal_handlers": "true", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_purge_ops": "10", 2024-09-16T13:28:25.226 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_truncate_ops": "128", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_blackhole": "false", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_snap": "true", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_caller_concurrency": "10", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_omap_check": "false", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_verify_split": "false", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_dump_file": "", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T13:28:25.227 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fadvise": "true", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fail_eio": "true", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_size": "128", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap": "false", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_inject_stall": "0", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_parallel": "false", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_trailing": "false", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_writeahead": "false", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_kill_at": "0", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T13:28:25.228 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T13:28:25.229 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_merge_threshold": "-10", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_odsync_write": "false", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend_path": "", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_threads": "2", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_punch_hole": "false", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T13:28:25.230 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_ops": "50", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_seek_data_hole": "false", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc": "false", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_splice": "false", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_multiple": "2", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_rand_factor": "20", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_update_to": "1000", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T13:28:25.231 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_xfs_extsize": "false", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_zfs_snap": "false", 2024-09-16T13:28:25.232 INFO:teuthology.orchestra.run.smithi076.stdout: "fio_dir": "/tmp/fio", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_allow_other": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_big_writes": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_debug": "false", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_default_permissions": "false", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_disable_pagecache": "false", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_max_write": "0", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_multithreaded": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_require_active_mds": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_set_user_groups": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_move": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_read": "true", 2024-09-16T13:28:25.233 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_write": "true", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_target_name": "ceph", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_file": "", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_inject_failure": "0", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_interval": "5", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "host": "smithi076", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T13:28:25.234 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "inject_early_sigterm": "false", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "jaeger_tracing_enable": "false", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_aio": "true", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_align_min_size": "65536", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_align": "true", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_size": "4096", 2024-09-16T13:28:25.235 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_dio": "true", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_discard": "false", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_force_aio": "false", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_ignore_corruption": "false", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_entries": "100", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_replay_from": "0", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_write_header_frequency": "0", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_zero_on_create": "false", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prefetch_periods": "10", 2024-09-16T13:28:25.236 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prezero_periods": "5", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_write_head_interval": "15", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "key": "", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "keyfile": "", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_backend": "rocksdb", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_bytes": "67108864", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_ops": "512", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_onode_map_size": "1024", 2024-09-16T13:28:25.237 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_transaction": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_block_size": "65536", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_bloom_size": "0", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_cache_size": "536870912", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_compact_on_mount": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_compression": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_log": "", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_max_open_files": "0", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_paranoid": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "librados_thread_count": "2", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep_force_backtrace": "false", 2024-09-16T13:28:25.238 INFO:teuthology.orchestra.run.smithi076.stdout: "log_coarse_timestamps": "true", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_flush_on_exit": "false", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_port": "12201", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_new": "1000", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_recent": "10000", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stderr_prefix": "", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": "true", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_graylog": "false", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_journald": "false", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_stderr": "false", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_syslog": "false", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T13:28:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_action_on_write_error": "1", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_alternate_name_max": "8192", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_asio_thread_count": "2", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_export_pin": "true", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_interval": "10", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max": "-1", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max_until": "-1", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_merge_size": "50", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T13:28:25.240 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_mode": "0", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_bits": "3", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_size": "10000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T13:28:25.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_mid": "0.700000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_interval": "1", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T13:28:25.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_auth_pins": "false", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_frag": "false", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_scatterstat": "false", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_subtrees": "false", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_default_dir_hash": "2", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_defer_session_stale": "true", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T13:28:25.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_max_entries": "0", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_early_reply": "true", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enable_op_tracker": "true", 2024-09-16T13:28:25.244 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enforce_unique_name": "true", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_health_dummy": "false", 2024-09-16T13:28:25.245 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_join_fs": "", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_journal_format": "1", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_create_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_export_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_import_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_link_at": "0", 2024-09-16T13:28:25.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_openc_at": "0", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_rename_at": "0", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_events": "-1", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_segments": "128", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_pause": "false", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_segment_size": "0", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_requests": "100000", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_export_size": "20971520", 2024-09-16T13:28:25.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_file_recover": "32", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_files": "64", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops": "8192", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_metrics_update_interval": "2", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_per_client": "100", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_numa_node": "-1", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T13:28:25.248 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_duration": "600", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_size": "20", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_log_threshold": "5", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_grace": "15", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_interval": "5", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_caps": "30000", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T13:28:25.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_interval": "1.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_gid": "0", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_uid": "0", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T13:28:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_shutdown_check": "0", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_skip_ino": "0", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_min_uid": "0", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_rstat": "false", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_symlink_recovery": "true", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_exports": "0", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_fragments": "0", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_tick_interval": "5.000000", 2024-09-16T13:28:25.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_valgrind_exit": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_backtrace": "1", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_scatter": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_sessions": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool_debug": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_set": "false", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_size": "65536", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_bytes": "134217728", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_messages": "512", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T13:28:25.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_messages": "128", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_messages": "128", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_messages": "8192", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_pool": "true", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_standby_modules": "true", 2024-09-16T13:28:25.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_period": "5", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_threshold": "5", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_tick_period": "2", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_delete": "true", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_bytes": "104857600", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T13:28:25.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_level": "debug", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T13:28:25.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_start": "false", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_trim": "true", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cpu_threads": "4", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T13:28:25.256 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_crit": "5", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_warn": "5", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_json": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_extra_checks": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_pacific": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_quincy": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T13:28:25.257 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_default_strategy": "1", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_timeout": "5.000000", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_enable_op_tracker": "true", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_fake_pool_delete": "false", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_force_quorum_join": "false", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_log_update_period": "5", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_max_detail": "50", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog": "true", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T13:28:25.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host": "172.21.15.76,[v2:172.21.15.76:3301,v1:172.21.15.76:6790],172.21.15.163", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host_override": "", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_initial_members": "", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease": "5.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_full_interval": "50", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_max": "10000", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_max_summary": "50", 2024-09-16T13:28:25.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_epochs": "500", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_osd": "10000", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_autotune": "true", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_target": "2147483648", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T13:28:25.260 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_digest_period": "5", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_complaint_time": "30", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_duration": "600", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_size": "20", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_log_threshold": "5", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T13:28:25.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size": "500", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T13:28:25.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_report_timeout": "900", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T13:28:25.263 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T13:28:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_osds": "4", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_interval": "86400", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_max_keys": "100", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_timeout": "300", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_session_timeout": "300", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_smart_report_timeout": "5", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T13:28:25.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_size": "4", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_debug": "false", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_tick_interval": "5", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T13:28:25.266 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_older_version": "true", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T13:28:25.267 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": "", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_op_threads": "3", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_cm": "false", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_device_name": "", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T13:28:25.268 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_sl": "3", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_type": "ib", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_reap_threshold": "5", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_before_connect": "false", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv4": "true", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv6": "false", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr1": "true", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr2": "true", 2024-09-16T13:28:25.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_max": "7568", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_min": "6800", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_count": "3", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_delay": "5", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_client": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mds": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mgr": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mon": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_osd": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_client_mode": "crc secure", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_type": "", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_compress_secure": "false", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T13:28:25.270 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_data": "true", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_header": "true", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bug": "true", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_old_message": "true", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T13:28:25.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hugepages": "", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_lro": "true", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_pmd": "", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_port_id": "0", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_on_send": "false", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T13:28:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_type": "", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_socket_failures": "5000", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_accept_failures": "4", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_backoff": "15.000000", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compress_mode": "none", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_min_cost": "65536", 2024-09-16T13:28:25.273 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_public_type": "", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_service_mode": "crc secure", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_nodelay": "true", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_type": "async+posix", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "no_config_file": "false", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_ops": "1024", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T13:28:25.274 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_timeout": "10.000000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "objectstore_blackhole": "false", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "openssl_engine_opts": "", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_ops": "4", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_slop": "0.020000", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T13:28:25.275 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_asio_thread_count": "2", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_max": "512", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_min": "64", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_peering": "false", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_beacon_report_interval": "300", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_duration": "30", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T13:28:25.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_blkin_trace_all": "false", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_update_on_start": "true", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_cap": "256", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_op_priority": "63", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_watch_timeout": "30", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_command_max_records": "256", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_compact_on_start": "false", 2024-09-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_on_start": "true", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T13:28:25.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_op_order": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_shutdown": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T13:28:25.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_notify_timeout": "30", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_enable_op_tracker": "true", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T13:28:25.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_info": "true", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown": "true", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_function_tracing": "false", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_grace": "20", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_interval": "6", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_stale": "600", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_size": "5120", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_kill_backfill_at": "0", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_cache_size": "50", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_dedup": "true", 2024-09-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max": "40", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_name_len": "100", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_size": "0", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_backfills": "1", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_count": "5", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_period": "600", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_name_len": "2048", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_size": "134217728", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pgls": "1024", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_cost": "8388608", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_objects": "10", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_scrubs": "3", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_size": "90", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_profile": "balanced", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_base": "805306368", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target": "4294967296", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target_autotune": "false", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_recovery_priority": "0", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_interval": "5", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_cache_shards": "32", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_node": "-1", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objecter_finishers": "1", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore": "bluestore", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_fuse": "false", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_tracing": "false", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_duration": "600", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_size": "20", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_log_threshold": "5", 2024-09-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards": "0", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_timeout": "15", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_open_classes_on_start": "true", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_os_flags": "0", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_peering_op_priority": "255", 2024-09-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_priority": "5", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flags": "0", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_min_size": "0", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_size": "2", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_type": "replicated", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_cost": "20971520", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active": "0", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_priority": "3", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_priority": "5", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_repair_during_recovery": "false", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T13:28:25.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_cost": "52428800", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_hour": "0", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T13:28:25.293 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_priority": "5", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_skip_data_digest": "false", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_smart_report_timeout": "5", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_priority": "5", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_transaction_size": "30", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tracing": "false", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_use_stale_snap": "false", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_kill_at": "0", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_max_join_drift": "10", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min": "500", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min_wait": "0.050000", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_max": "500", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_min": "250", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_stash_full_interval": "25", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_max": "500", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_min": "250", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "perf": "true", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "-", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrv": "", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "public_bind_addr": "-", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network": "", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network_interface": "", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "qat_compressor_enabled": "false", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_mon_op_timeout": "0", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_osd_op_timeout": "0", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_tracing": "false", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_atime_update_interval": "60", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache": "true", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_policy": "writearound", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_size": "33554432", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_compression_hint": "none", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_clone_format": "auto", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_data_pool": "", 2024-09-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_features": "61", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_format": "2", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_map_options": "", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_order": "22", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_pool": "rbd", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_count": "0", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler": "simple", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_order": "24", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_pool": "", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_splay_width": "4", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_threads": "1", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_plugins": "", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T13:28:25.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_tracing": "false", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_names": "true", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_pool": "true", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_base_url": "", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_log_level": "", 2024-09-16T13:28:25.305 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_admin_entry": "admin", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_backend_store": "rados", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_barbican_url": "", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_beast_enable_async": "true", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_enabled": "true", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_lru_size": "10000", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_content_length_compat": "false", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T13:28:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cross_domain_policy": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T13:28:25.307 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T13:28:25.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T13:28:25.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_window": "30", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T13:28:25.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T13:28:25.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_name": "", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_s3website_name": "", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_ops_log": "false", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T13:28:25.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_static_website": "false", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_usage_log": "false", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_expose_bucket": "false", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_extended_http_attrs": "", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_objs": "32", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T13:28:25.313 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_init_timeout": "300", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password": "", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T13:28:25.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_project": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_user": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_api_version": "2", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T13:28:25.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_url": "", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_objs": "32", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_worker": "3", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_uri": "", 2024-09-16T13:28:25.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_http_headers": "", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_size": "0", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_listing_results": "1000", 2024-09-16T13:28:25.317 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_notify_retries": "10", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T13:28:25.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_control_oids": "8", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_numa_node": "-1", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T13:28:25.319 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_tracing": "false", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_token": "", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_url": "", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_file_path": "", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_rados": "true", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T13:28:25.320 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_continue": "true", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_tracing": "false", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm": "", 2024-09-16T13:28:25.321 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm_id": "", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region": "", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_request_uri": "", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_resolve_cname": "false", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T13:28:25.322 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_run_sync_thread": "true", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_script_uri": "", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_service_provider_name": "", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_id": "", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_secret": "", 2024-09-16T13:28:25.323 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_entry": "sts", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_key": "sts", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_url": "", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_custom_header": "", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_need_stats": "true", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_tenant_name": "", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url": "", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T13:28:25.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_lease_period": "120", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_thread_pool_size": "512", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_comment": "", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_createby": "", 2024-09-16T13:28:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_encoding": "", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_flag": "false", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_origin": "", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_tracker": "", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_shards": "32", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_use_opa_authz": "false", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T13:28:25.326 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_unique_email": "true", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_verify_ssl": "true", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone": "", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone_id": "", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup": "", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup_id": "", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_block_size": "4096", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T13:28:25.327 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T13:28:25.328 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_partition_filters": "false", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_perf": "false", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "run_dir": "/var/run/ceph", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_block_create": "true", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_device_size": "107374182400", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T13:28:25.329 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "seastore_segment_size": "67108864", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "setgroup": "", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser": "", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser_match_path": "", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "thp": "false", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_default_timeout": "60", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout: "throttler_perf_counter": "true" 2024-09-16T13:28:25.330 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:25.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T13:28:25.439 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a: {} 2024-09-16T13:28:25.439 INFO:teuthology.orchestra.run.smithi076.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:28:25.448 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b: {} 2024-09-16T13:28:25.448 INFO:teuthology.orchestra.run.smithi076.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:28:25.456 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c: {} 2024-09-16T13:28:25.456 INFO:teuthology.orchestra.run.smithi076.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:28:25.467 DEBUG:teuthology.orchestra.run.smithi076:> type ceph-objectstore-tool 2024-09-16T13:28:25.481 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T13:28:25.481 DEBUG:teuthology.orchestra.run.smithi163:> type ceph-objectstore-tool 2024-09-16T13:28:25.500 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T13:28:25.500 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-09-16T13:28:25.510 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-09-16T13:28:25.510 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T13:28:25.510 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:25.511 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:28:25.511 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T13:28:25.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:28:25.511 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T13:28:25.512 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T13:28:25.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:25.512 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T13:28:25.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:25.513 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T13:28:25.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:28:25.513 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-09-16T13:28:25.522 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:25.520+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:25.551 INFO:teuthology.orchestra.run.smithi076.stdout:mapped writes DISABLED 2024-09-16T13:28:25.551 INFO:teuthology.orchestra.run.smithi076.stdout:mapped reads DISABLED 2024-09-16T13:28:25.551 INFO:teuthology.orchestra.run.smithi076.stdout:Seed set to 2515 2024-09-16T13:28:25.613 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:25.611+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:25.631 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:25.631 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:25.631 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:25.631 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:25.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:25.683 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:25.683 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:25.700 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:25.701 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:25.714 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:25.712+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:25.718 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:25.718 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:25.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:25.733 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:25.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:25.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:25.765 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:25.765 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:25.782 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:25.782 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:25.783 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:25.784 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:25.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:25.798 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:25.798 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:25.809 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:25.809 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:25.809 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 31, 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:28:23.951903+0000", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:27:52.988177+0000", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:28:25.810 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 3, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "jewel", 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:28:25.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:25.812 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:25.813 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:28:25.814 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:28:25.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:28:25.816 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:25.817 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:25.818 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:28:25.819 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:28:25.820 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:25.821 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:28:25.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:28:25.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:25.824 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:25.825 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:25.826 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:28:25.827 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 24, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 24, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:28:25.838 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 19, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:28:25.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.843 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.844 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 24, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:28:25.848 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:28:25.849 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:25.850 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 24, 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.853 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.855 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 19, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:28:25.859 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:28:25.860 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 19, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:28:25.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 19, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:25.866 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.867 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:28:25.870 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.871 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.872 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:28:25.873 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:28:25.875 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:28:25.876 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [], 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:28:25.877 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [], 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:25.878 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:25.880 INFO:tasks.thrashosds.thrasher:j is {'epoch': 31, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:28:23.951903+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:27:52.988177+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:28:25.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:25.882 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:25.880+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:25.899 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:25.899 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:25.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:25.976 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:28:25.983 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:25.981+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:26.058 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.058 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:26.084 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:26.082+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:26.168 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.168 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:26.181 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:26.182 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:26.185 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:26.183+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:26.194 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-16T13:28:25.968682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381468,"kb_used_data":2568,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752593252,"statfs":{"total":773094113280,"available":770655490048,"internally_reserved":0,"allocated":2629632,"data_stored":2150170,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.128211"},"pg_stats":[{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.932229+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:20.932229+0000","last_peered":"2024-09-16T13:28:20.932229+0000","last_clean":"2024-09-16T13:28:20.932229+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:20.932229+0000","last_undegraded":"2024-09-16T13:28:20.932229+0000","last_fullsized":"2024-09-16T13:28:20.932229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934822+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934822+0000","last_peered":"2024-09-16T13:28:20.934822+0000","last_clean":"2024-09-16T13:28:20.934822+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:20.934822+0000","last_undegraded":"2024-09-16T13:28:20.934822+0000","last_fullsized":"2024-09-16T13:28:20.934822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:22.849832+0000","last_change":"2024-09-16T13:28:12.959294+0000","last_active":"2024-09-16T13:28:22.849832+0000","last_peered":"2024-09-16T13:28:22.849832+0000","last_clean":"2024-09-16T13:28:22.849832+0000","last_became_active":"2024-09-16T13:28:10.306929+0000","last_became_peered":"2024-09-16T13:28:10.306929+0000","last_unstale":"2024-09-16T13:28:22.849832+0000","last_undegraded":"2024-09-16T13:28:22.849832+0000","last_fullsized":"2024-09-16T13:28:22.849832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:42.738385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034515700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934809+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:20.934809+0000","last_peered":"2024-09-16T13:28:20.934809+0000","last_clean":"2024-09-16T13:28:20.934809+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:20.934809+0000","last_undegraded":"2024-09-16T13:28:20.934809+0000","last_fullsized":"2024-09-16T13:28:20.934809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.947717+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:21.947717+0000","last_peered":"2024-09-16T13:28:21.947717+0000","last_clean":"2024-09-16T13:28:21.947717+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:21.947717+0000","last_undegraded":"2024-09-16T13:28:21.947717+0000","last_fullsized":"2024-09-16T13:28:21.947717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:28:23.378139+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:23.378139+0000","last_peered":"2024-09-16T13:28:23.378139+0000","last_clean":"2024-09-16T13:28:23.378139+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:23.378139+0000","last_undegraded":"2024-09-16T13:28:23.378139+0000","last_fullsized":"2024-09-16T13:28:23.378139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T13:28:20.934859+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:28:20.934859+0000","last_peered":"2024-09-16T13:28:20.934859+0000","last_clean":"2024-09-16T13:28:20.934859+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:28:20.934859+0000","last_undegraded":"2024-09-16T13:28:20.934859+0000","last_fullsized":"2024-09-16T13:28:20.934859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.994776+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:21.994776+0000","last_peered":"2024-09-16T13:28:21.994776+0000","last_clean":"2024-09-16T13:28:21.994776+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:21.994776+0000","last_undegraded":"2024-09-16T13:28:21.994776+0000","last_fullsized":"2024-09-16T13:28:21.994776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.949880+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:28:21.949880+0000","last_peered":"2024-09-16T13:28:21.949880+0000","last_clean":"2024-09-16T13:28:21.949880+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:28:21.949880+0000","last_undegraded":"2024-09-16T13:28:21.949880+0000","last_fullsized":"2024-09-16T13:28:21.949880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:31.394402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:22.849834+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:22.849834+0000","last_peered":"2024-09-16T13:28:22.849834+0000","last_clean":"2024-09-16T13:28:22.849834+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:22.849834+0000","last_undegraded":"2024-09-16T13:28:22.849834+0000","last_fullsized":"2024-09-16T13:28:22.849834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:21.949052+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:21.949052+0000","last_peered":"2024-09-16T13:28:21.949052+0000","last_clean":"2024-09-16T13:28:21.949052+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:21.949052+0000","last_undegraded":"2024-09-16T13:28:21.949052+0000","last_fullsized":"2024-09-16T13:28:21.949052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:52.853771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:28:23.378114+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:23.378114+0000","last_peered":"2024-09-16T13:28:23.378114+0000","last_clean":"2024-09-16T13:28:23.378114+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:23.378114+0000","last_undegraded":"2024-09-16T13:28:23.378114+0000","last_fullsized":"2024-09-16T13:28:23.378114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T13:28:22.849932+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:22.849932+0000","last_peered":"2024-09-16T13:28:22.849932+0000","last_clean":"2024-09-16T13:28:22.849932+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:22.849932+0000","last_undegraded":"2024-09-16T13:28:22.849932+0000","last_fullsized":"2024-09-16T13:28:22.849932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T13:28:25.959435+0000","last_change":"2024-09-16T13:28:25.959435+0000","last_active":"2024-09-16T13:28:25.959435+0000","last_peered":"2024-09-16T13:28:25.959435+0000","last_clean":"2024-09-16T13:28:25.959435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T13:28:25.959435+0000","last_undegraded":"2024-09-16T13:28:25.959435+0000","last_fullsized":"2024-09-16T13:28:25.959435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297728,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074112,"statfs":{"total":96636764160,"available":96331890688,"internally_reserved":0,"allocated":376832,"data_stored":714618,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297696,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074144,"statfs":{"total":96636764160,"available":96331923456,"internally_reserved":0,"allocated":335872,"data_stored":130420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":319488,"data_stored":124250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":319488,"data_stored":124250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":294912,"data_stored":111948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":319488,"data_stored":124250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297648,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074192,"statfs":{"total":96636764160,"available":96331972608,"internally_reserved":0,"allocated":299008,"data_stored":111967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":364544,"data_stored":708467,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:26.196 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [6, 1] 2024-09-16T13:28:26.196 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.5', '6', '1'] 2024-09-16T13:28:26.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.5 6 1 2024-09-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.286 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:26.284+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:26.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:26.387 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:26.385+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:26.429 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.430 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:26.487 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:26.486+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:26.538 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.539 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:26.591 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:26.589+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:26.645 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.645 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:26.690 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:26.689+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:26.791 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:26.789+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:26.793 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.793 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:26.891 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:26.890+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:26.940 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:26.940 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:26.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:26.975 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.5 pg_upmap mapping to [6,1] 2024-09-16T13:28:26.992 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:26.990+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:27.078 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.078 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:27.093 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:27.091+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:27.194 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.194 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.194 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:27.192+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:27.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:27.295 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:27.293+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:27.300 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.300 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:27.396 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:27.394+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:27.405 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.405 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:27.497 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:27.495+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:27.512 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.513 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:27.607 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:27.605+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:27.663 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.663 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:27.713 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:27.711+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:27.800 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:27.798+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:27.811 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.812 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:27.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:27.900 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:27.898+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:27.923 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:27.923 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:27.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:27.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:28:28.003 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:28.003 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:28.003 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:28.001+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:28.019 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:28.019 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:28.034 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:28.034 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:28.049 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:28.049 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:28.065 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:28.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:28.074 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:28.074 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:28.080 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:28.080 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:28.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:28.099 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:28.101 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:28.100+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:28.116 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:28.116 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:28.184 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:28.184 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:28.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:28.184 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [] 2024-09-16T13:28:28.184 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:28.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:28.236 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:28.234+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:28.293 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:28.294 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:28.294 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:28.294 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:28.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:28.303 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:28.302+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:28.404 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:28.403+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:28.440 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:28.441 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:28.441 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:28.441 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:28.441 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:28.449 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:28.505 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:28.503+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.259391026999999, 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:28:28.553 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:28:28.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:28:28.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.559 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.862040+0000", 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.686551655999999, 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074038560000000003, 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 22 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.560 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862040+0000", 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862040+0000", 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862040+0000", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-07 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862098+0000", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7839e-05 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862105+0000", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2360000000000001e-06 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862108+0000", 2024-09-16T13:28:28.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8399999999999997e-06 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.863585+0000", 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014767249999999999 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.863665+0000", 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9930000000000002e-05 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.863778+0000", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011347499999999999 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.866199+0000", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024206969999999999 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869391+0000", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031920350000000002 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869423+0000", 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1903999999999998e-05 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869444+0000", 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.069e-05 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:23 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.862113+0000", 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.686478469000001, 2024-09-16T13:28:28.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073819649999999999, 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 23 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862113+0000", 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862113+0000", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862114+0000", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.32e-07 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862180+0000", 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6519000000000005e-05 2024-09-16T13:28:28.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862186+0000", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.728e-06 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.862189+0000", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.422e-06 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.864413+0000", 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022238470000000002 2024-09-16T13:28:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.864466+0000", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2531000000000002e-05 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.864583+0000", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011704700000000001 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.866302+0000", 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001719663 2024-09-16T13:28:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869451+0000", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031481930000000001 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869469+0000", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8830000000000001e-05 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.869495+0000", 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5752999999999999e-05 2024-09-16T13:28:28.570 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.558040829999999, 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.571 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557918434000001, 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.576 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557806046, 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:28:28.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:28:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557743232, 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:28:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:28:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:28:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.592 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557595802000002, 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:28.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:28:28.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:28:28.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:28:28.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:28:28.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557580618999999, 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:28:28.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:28.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:28:28.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.603 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557559571999999, 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:28:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:28:28.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.609 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557472092000001, 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:28:28.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557449583, 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.615 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:28:28.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:28:28.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.620 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557403256000001, 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.621 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:28:28.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:28:28.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557236498999998, 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:28.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:28:28.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.632 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.557221733999999, 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:28:28.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336631+0000", 2024-09-16T13:28:28.638 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.211960166000001, 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.007622088, 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336631+0000", 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336631+0000", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336632+0000", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1199999999999999e-07 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336675+0000", 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:28:28.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336682+0000", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.798e-06 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336686+0000", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.341278+0000", 2024-09-16T13:28:28.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045922929999999999 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.341351+0000", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2991000000000006e-05 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.341469+0000", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011817 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342817+0000", 2024-09-16T13:28:28.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013484759999999999 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344204+0000", 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001386155 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344231+0000", 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7365999999999999e-05 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344254+0000", 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2520000000000001e-05 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.211899417000001, 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.644 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:28.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:28:28.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.211861739, 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:28:28.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.655 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.211838925999999, 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.656 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:28:28.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:28:28.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:28:28.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.211753713, 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:28.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:28.667 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:28.668 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:28.666+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:28.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:28.767 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:28.767 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:28.767 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:28.767 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:28.769 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:28.767+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:28.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:28.870 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:28.869+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:28.913 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:28.914 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:28.914 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:28.914 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:28.914 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:28.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:28.972 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:28.970+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.029 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:9 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'4, mlcod=17'2)", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.845597+0000", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.181144038999999, 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017024270000000001, 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845597+0000", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845597+0000", 2024-09-16T13:28:29.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845598+0000", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6499999999999998e-07 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845604+0000", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4320000000000004e-06 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845617+0000", 2024-09-16T13:28:29.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2599000000000001e-05 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845623+0000", 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5589999999999996e-06 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845688+0000", 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5394000000000004e-05 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.845703+0000", 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4808e-05 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.847263+0000", 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001560325 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.847300+0000", 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6445000000000001e-05 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:40 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'22, mlcod=17'19)", 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.991457+0000", 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.035284243, 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029783980000000002, 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991457+0000", 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991457+0000", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991457+0000", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5499999999999998e-07 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991465+0000", 2024-09-16T13:28:29.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9410000000000007e-06 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991476+0000", 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0414999999999999e-05 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991480+0000", 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3800000000000004e-06 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991514+0000", 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3887000000000003e-05 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991528+0000", 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3699999999999999e-05 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994411+0000", 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028830129999999998 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994435+0000", 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4507000000000001e-05 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:41 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'23, mlcod=17'19)", 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.991952+0000", 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.034789315000001, 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049349809999999997, 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.038 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991952+0000", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991952+0000", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.991959+0000", 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.968e-06 2024-09-16T13:28:29.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992015+0000", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6019e-05 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992023+0000", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9829999999999996e-06 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992027+0000", 2024-09-16T13:28:29.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1790000000000001e-06 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992055+0000", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8442e-05 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992067+0000", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1698999999999999e-05 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996864+0000", 2024-09-16T13:28:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047973060000000003 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996887+0000", 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2385e-05 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:42 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'24, mlcod=17'20)", 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.992911+0000", 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.033830182999999, 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039929450000000003, 2024-09-16T13:28:29.042 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992911+0000", 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992911+0000", 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992911+0000", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992918+0000", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6220000000000003e-06 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992925+0000", 2024-09-16T13:28:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7530000000000004e-06 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992929+0000", 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1309999999999999e-06 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992960+0000", 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1245000000000003e-05 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.992974+0000", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3569e-05 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996888+0000", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039142919999999998 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996904+0000", 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5880999999999999e-05 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:43 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'25, mlcod=17'20)", 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.993340+0000", 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.033400844999999, 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003576367, 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993340+0000", 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993340+0000", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993341+0000", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993346+0000", 2024-09-16T13:28:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6919999999999999e-06 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993353+0000", 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1870000000000002e-06 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993356+0000", 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7400000000000002e-06 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993384+0000", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8014999999999999e-05 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993397+0000", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-05 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996906+0000", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00350892 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.996917+0000", 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1031e-05 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:44 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'26, mlcod=17'20)", 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.993883+0000", 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.032858140999998, 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034551740000000001, 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993883+0000", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993883+0000", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993883+0000", 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993945+0000", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1377999999999994e-05 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993954+0000", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0999999999999993e-06 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993958+0000", 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.993988+0000", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.987e-05 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994001+0000", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2943e-05 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033098110000000002 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997338+0000", 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7645e-05 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:45 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'27, mlcod=17'20)", 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.994726+0000", 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.032015611999999, 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002627824, 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994726+0000", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994726+0000", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994726+0000", 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:28:29.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994736+0000", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0067e-05 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994743+0000", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.002e-06 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994747+0000", 2024-09-16T13:28:29.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9480000000000001e-06 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994778+0000", 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0907000000000001e-05 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.994790+0000", 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2243e-05 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997339+0000", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025490320000000001 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997353+0000", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4225e-05 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:46 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'28, mlcod=17'20)", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.995150+0000", 2024-09-16T13:28:29.059 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.031591549000002, 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022173459999999998, 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995150+0000", 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995150+0000", 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995150+0000", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995155+0000", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9420000000000003e-06 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995161+0000", 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.331e-06 2024-09-16T13:28:29.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995168+0000", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9689999999999997e-06 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995199+0000", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0519999999999999e-05 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995210+0000", 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1028e-05 2024-09-16T13:28:29.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997354+0000", 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002144211 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997367+0000", 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2935000000000001e-05 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:47 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'29, mlcod=17'21)", 2024-09-16T13:28:29.063 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:05.995760+0000", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.030981484000002, 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001617945, 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995760+0000", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995760+0000", 2024-09-16T13:28:29.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995760+0000", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995822+0000", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1843999999999999e-05 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995829+0000", 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.311e-06 2024-09-16T13:28:29.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995834+0000", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5820000000000003e-06 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995865+0000", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1167000000000002e-05 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.995877+0000", 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1702000000000001e-05 2024-09-16T13:28:29.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997368+0000", 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014909859999999999 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:05.997378+0000", 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9529999999999994e-06 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:56 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'38, mlcod=17'33)", 2024-09-16T13:28:29.067 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.003588+0000", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.023153173000001, 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027489010000000002, 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003588+0000", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003588+0000", 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003588+0000", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003654+0000", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5686999999999996e-05 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003661+0000", 2024-09-16T13:28:29.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5200000000000003e-06 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003664+0000", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.145e-06 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003690+0000", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6648e-05 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.003702+0000", 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1667e-05 2024-09-16T13:28:29.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.006307+0000", 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026051429999999999 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.006337+0000", 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9689e-05 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.071 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:57 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'39, mlcod=17'37)", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.007112+0000", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.019629629000001, 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019611820000000001, 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007112+0000", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007112+0000", 2024-09-16T13:28:29.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007112+0000", 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007117+0000", 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6299999999999997e-06 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007127+0000", 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0494e-05 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007131+0000", 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0149999999999997e-06 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007169+0000", 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7449e-05 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.007183+0000", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3777e-05 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.009046+0000", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018635660000000001 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.009073+0000", 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6591e-05 2024-09-16T13:28:29.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:71 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'49, mlcod=18'44)", 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.340843+0000", 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.685898439999999, 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001570901, 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340843+0000", 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340843+0000", 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340843+0000", 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7400000000000003e-07 2024-09-16T13:28:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340848+0000", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.853e-06 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340858+0000", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7010000000000008e-06 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340862+0000", 2024-09-16T13:28:29.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6669999999999998e-06 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340896+0000", 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4440999999999998e-05 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.340913+0000", 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6701000000000002e-05 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342370+0000", 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014575619999999999 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342414+0000", 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3302e-05 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:72 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'50, mlcod=18'46)", 2024-09-16T13:28:29.080 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.341512+0000", 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.685228689999999, 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015746320000000001, 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341512+0000", 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341512+0000", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341513+0000", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7399999999999998e-07 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341574+0000", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0674000000000002e-05 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341584+0000", 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0057e-05 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341587+0000", 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6270000000000002e-06 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341617+0000", 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9662000000000002e-05 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.341635+0000", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8332e-05 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343065+0000", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001429257 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343087+0000", 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2549e-05 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:73 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'51, mlcod=18'46)", 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.342446+0000", 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.684294858000001, 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003501288, 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342446+0000", 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342446+0000", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342447+0000", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342451+0000", 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4340000000000002e-06 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342471+0000", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9755999999999999e-05 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342475+0000", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9559999999999999e-06 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342504+0000", 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8966999999999999e-05 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.342516+0000", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2275e-05 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.345924+0000", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034078870000000001 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.345948+0000", 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3408e-05 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:74 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'52, mlcod=18'48)", 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.343087+0000", 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.683653677999999, 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052697789999999996, 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343087+0000", 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343087+0000", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343088+0000", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6199999999999999e-07 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343091+0000", 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9569999999999998e-06 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343097+0000", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2419999999999997e-06 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343100+0000", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343127+0000", 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6832e-05 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343139+0000", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2516e-05 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.348325+0000", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051854199999999996 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.348357+0000", 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2441000000000003e-05 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:75 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'53, mlcod=18'48)", 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.343615+0000", 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.683125832000002, 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075448090000000004, 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343615+0000", 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343615+0000", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343616+0000", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343644+0000", 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8050000000000001e-05 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343654+0000", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343658+0000", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9439999999999998e-06 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343687+0000", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8878999999999999e-05 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.343698+0000", 2024-09-16T13:28:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0864e-05 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.351135+0000", 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074373920000000001 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.351160+0000", 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5038999999999999e-05 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:76 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'54, mlcod=18'49)", 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.344532+0000", 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.682209102000002, 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006648607, 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344532+0000", 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344532+0000", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344533+0000", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5400000000000001e-07 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344538+0000", 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4750000000000001e-06 2024-09-16T13:28:29.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344548+0000", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7540000000000001e-06 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.344551+0000", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6610000000000002e-06 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.345974+0000", 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001422956 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.345987+0000", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2249000000000001e-05 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.351162+0000", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051748710000000002 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.351181+0000", 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9086999999999999e-05 2024-09-16T13:28:29.100 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:92 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'66, mlcod=18'62)", 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.801553+0000", 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.225188143, 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035487890000000001, 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801553+0000", 2024-09-16T13:28:29.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801553+0000", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801553+0000", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7399999999999999e-07 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801558+0000", 2024-09-16T13:28:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0320000000000003e-06 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801566+0000", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6939999999999995e-06 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801571+0000", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6229999999999996e-06 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801597+0000", 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5950000000000001e-05 2024-09-16T13:28:29.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801608+0000", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1109e-05 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805079+0000", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034715530000000001 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805102+0000", 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2453999999999999e-05 2024-09-16T13:28:29.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:93 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'67, mlcod=18'62)", 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.801978+0000", 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.224762729999998, 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034082370000000002, 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801978+0000", 2024-09-16T13:28:29.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801978+0000", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.801979+0000", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4499999999999997e-07 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802042+0000", 2024-09-16T13:28:29.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2613999999999994e-05 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802049+0000", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5100000000000001e-06 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802052+0000", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802077+0000", 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5038999999999999e-05 2024-09-16T13:28:29.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802091+0000", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3292000000000001e-05 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805368+0000", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032773120000000001 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805387+0000", 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8606000000000001e-05 2024-09-16T13:28:29.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4181.0:94 1.0 e18/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'68, mlcod=18'63)", 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:06.802847+0000", 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.223893819000001, 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027748439999999998, 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802847+0000", 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802847+0000", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802848+0000", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1399999999999997e-07 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802851+0000", 2024-09-16T13:28:29.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9689999999999999e-06 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802857+0000", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6409999999999996e-06 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.802862+0000", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5859999999999998e-06 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805116+0000", 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022543440000000001 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805132+0000", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5948999999999999e-05 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805602+0000", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046989900000000002 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:06.805622+0000", 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0041999999999999e-05 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.112 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.113 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.113 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:29.114 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:29.113+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:29.214 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:29.213+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:29.253 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:29.253 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:29.315 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:29.313+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:29.402 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.402 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:29.402 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:29.402 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:29.402 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:29.416 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:29.415+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:5 2.2 e21/19 2:5cea7035:::rbd_info:head v 21'1, mlcod=0'0)", 2024-09-16T13:28:29.510 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.200466084999999, 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011418870000000001, 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308444+0000", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308452+0000", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7539999999999995e-06 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308482+0000", 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9589000000000001e-05 2024-09-16T13:28:29.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308497+0000", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4671999999999999e-05 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308579+0000", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2148e-05 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308608+0000", 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9612e-05 2024-09-16T13:28:29.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.309541+0000", 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00093304800000000002 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.309585+0000", 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3538000000000002e-05 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.514 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:7 2.2 e22/19 2:5cea7035:::rbd_info:head v 22'2, mlcod=0'0)", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.191410297000001, 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00083179599999999999, 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.16e-07 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317504+0000", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7539999999999997e-06 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317518+0000", 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4516e-05 2024-09-16T13:28:29.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317527+0000", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0180000000000007e-06 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317583+0000", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5659999999999999e-05 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317619+0000", 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6365000000000002e-05 2024-09-16T13:28:29.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.318296+0000", 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067628100000000004 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.318331+0000", 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4986e-05 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:10 4.1c e35/32 4:3831950c:::rbd_directory:head v 35'1, mlcod=0'0)", 2024-09-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.52288926300000005, 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054675700000000004, 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986025+0000", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3020000000000002e-06 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986041+0000", 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5993000000000001e-05 2024-09-16T13:28:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986047+0000", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986086+0000", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8921999999999998e-05 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986105+0000", 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9364999999999998e-05 2024-09-16T13:28:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986537+0000", 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043148900000000002 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986567+0000", 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9963000000000001e-05 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.522 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:29.524 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:29.522+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:29.623 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.624 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:29.624 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:29.624 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.624 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:29.623+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:29.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:29.724 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:29.723+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:29.731 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.731 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:29.731 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:29.731 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:29.731 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.740 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:29.825 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:29.823+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:29.881 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:1 2.3 e20/19 2:c4c92e5a:::rbd_trash:head v 20'1, mlcod=0'0)", 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.567721876, 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045403689999999998, 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:29.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312624+0000", 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5340000000000001e-06 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312635+0000", 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1029000000000001e-05 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312643+0000", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0290000000000005e-06 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312677+0000", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3207000000000002e-05 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312699+0000", 2024-09-16T13:28:29.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2226e-05 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.317133+0000", 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044345080000000002 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.317160+0000", 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6548999999999999e-05 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:10 4.1c 4:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.89503411899999996, 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014440410000000001, 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:29.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 10 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985309+0000", 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.885e-06 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985316+0000", 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7669999999999998e-06 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985329+0000", 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2631e-05 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985337+0000", 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8380000000000008e-06 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985394+0000", 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6873000000000002e-05 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985539+0000", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144659 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985753+0000", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021416 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986407+0000", 2024-09-16T13:28:29.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00065385200000000001 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986663+0000", 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025650800000000002 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986711+0000", 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7620999999999997e-05 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986751+0000", 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0247000000000003e-05 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:10 4.1c e35/32 ondisk, result = 0)", 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.89374434400000002, 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016042399999999999, 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986598+0000", 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986602+0000", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986611+0000", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4760000000000004e-06 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986616+0000", 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3320000000000002e-06 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986649+0000", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3083000000000002e-05 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986658+0000", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.234e-06 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986757+0000", 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9628000000000002e-05 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.896 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:29.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:29.897 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:29.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:29.939 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:29.937+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:30.027 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:30.026+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:30.038 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:30.039 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:30.039 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:30.039 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:30.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:30.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:30.132 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:30.130+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:30.187 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:30.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:30.240 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:30.238+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:30.247 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:30.263 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:30.278 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:30.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:30.293 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:30.298 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.298 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:30.310 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:30.325 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:30.330 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:30.328+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:30.341 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:30.356 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:30.416 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.417 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.425 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:30.469 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:30.467+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:30.522 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.523 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.531 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:30.532 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:30.530+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:30.633 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:30.631+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:30.671 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.671 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:30.681 INFO:teuthology.orchestra.run.smithi076.stdout:skipping zero size read 2024-09-16T13:28:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:2 writesame 0x70bb040 thru 0x70c7808 data_size 0x35(0xc7c9 bytes) 2024-09-16T13:28:30.694 INFO:teuthology.orchestra.run.smithi076.stdout:4 write 0xf5c00f thru 0xf6b424 (0xf416 bytes) 2024-09-16T13:28:30.766 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:30.764+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:30.819 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.819 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:30.834 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:30.833+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:30.901 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:30.901 INFO:teuthology.orchestra.run.smithi076.stdout:7 write 0x6818a0a thru 0x682358f (0xab86 bytes) 2024-09-16T13:28:30.905 INFO:teuthology.orchestra.run.smithi076.stdout:8 read 0x223e3bb thru 0x2241f1b (0x3b61 bytes) 2024-09-16T13:28:30.905 INFO:teuthology.orchestra.run.smithi076.stdout:9 punch from 0x2881ca7 to 0x2884505, (0x285e bytes) 2024-09-16T13:28:30.905 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:30.905 INFO:teuthology.orchestra.run.smithi076.stdout:11 writesame 0x38e2d2e thru 0x38ed960 data_size 0x1(0xac33 bytes) 2024-09-16T13:28:30.936 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:30.933+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:30.970 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:30.970 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:30.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:30.991 INFO:teuthology.orchestra.run.smithi076.stdout:13 write 0x338e37c thru 0x339b722 (0xd3a7 bytes) 2024-09-16T13:28:30.996 INFO:teuthology.orchestra.run.smithi076.stdout:14 writesame 0x5ccf3e1 thru 0x5cd70fd data_size 0x1(0x7d1d bytes) 2024-09-16T13:28:30.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:28:31.044 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:31.042+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:31.059 INFO:teuthology.orchestra.run.smithi076.stdout:16 read 0xa347b7 thru 0xa423cc (0xdc16 bytes) 2024-09-16T13:28:31.059 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xc67ab03 2024-09-16T13:28:31.065 INFO:teuthology.orchestra.run.smithi076.stdout:18 trunc from 0x70c7809 to 0xc67ab03 2024-09-16T13:28:31.081 INFO:teuthology.orchestra.run.smithi076.stdout:20 writesame 0xda50e9d thru 0xda51055 data_size 0x3(0x1b9 bytes) 2024-09-16T13:28:31.081 INFO:teuthology.orchestra.run.smithi076.stdout:21 read 0x7372570 thru 0x73752ef (0x2d80 bytes) 2024-09-16T13:28:31.081 INFO:teuthology.orchestra.run.smithi076.stdout:23 read 0xb4951b7 thru 0xb49f0ce (0x9f18 bytes) 2024-09-16T13:28:31.094 INFO:teuthology.orchestra.run.smithi076.stdout:24 trunc from 0xda51056 to 0x3812222 2024-09-16T13:28:31.106 INFO:teuthology.orchestra.run.smithi076.stdout:25 punch from 0x172a5c5 to 0x1731506, (0x6f41 bytes) 2024-09-16T13:28:31.110 INFO:teuthology.orchestra.run.smithi076.stdout:26 trunc from 0x3812222 to 0x72947c6 2024-09-16T13:28:31.115 INFO:teuthology.orchestra.run.smithi076.stdout:27 read 0x124bcfe thru 0x1250022 (0x4325 bytes) 2024-09-16T13:28:31.116 INFO:teuthology.orchestra.run.smithi076.stdout:28 writesame 0x71c5dd3 thru 0x71cc073 data_size 0x1(0x62a1 bytes) 2024-09-16T13:28:31.125 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.125 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:31.141 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:31.139+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:31.169 INFO:teuthology.orchestra.run.smithi076.stdout:31 writesame 0x6d921b4 thru 0x6d9ab98 data_size 0x29(0x89e5 bytes) 2024-09-16T13:28:31.173 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.184 INFO:teuthology.orchestra.run.smithi076.stdout:33 write 0xb0ce6e2 thru 0xb0d0a53 (0x2372 bytes) 2024-09-16T13:28:31.188 INFO:teuthology.orchestra.run.smithi076.stdout:34 punch from 0x814b549 to 0x815285e, (0x7315 bytes) 2024-09-16T13:28:31.188 INFO:teuthology.orchestra.run.smithi076.stdout:35 write 0x723e003 thru 0x724cbc3 (0xebc1 bytes) 2024-09-16T13:28:31.192 INFO:teuthology.orchestra.run.smithi076.stdout:37 write 0x875d1b1 thru 0x8764ab9 (0x7909 bytes) 2024-09-16T13:28:31.196 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.196 INFO:teuthology.orchestra.run.smithi076.stdout:39 punch from 0xacfd6f to 0xad58a5, (0x5b36 bytes) 2024-09-16T13:28:31.196 INFO:teuthology.orchestra.run.smithi076.stdout:41 read 0x379d355 thru 0x37a957c (0xc228 bytes) 2024-09-16T13:28:31.196 INFO:teuthology.orchestra.run.smithi076.stdout:42 read 0x6624251 thru 0x6632671 (0xe421 bytes) 2024-09-16T13:28:31.205 INFO:teuthology.orchestra.run.smithi076.stdout:43 write 0xc73cb14 thru 0xc746a85 (0x9f72 bytes) 2024-09-16T13:28:31.209 INFO:teuthology.orchestra.run.smithi076.stdout:44 write 0x9ad49f3 thru 0x9ae0ccf (0xc2dd bytes) 2024-09-16T13:28:31.214 INFO:teuthology.orchestra.run.smithi076.stdout:45 read 0x7911c2d thru 0x791f7b0 (0xdb84 bytes) 2024-09-16T13:28:31.214 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.214 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xd51a49a 2024-09-16T13:28:31.215 INFO:teuthology.orchestra.run.smithi076.stdout:47 trunc from 0xc746a86 to 0xd51a49a 2024-09-16T13:28:31.222 INFO:teuthology.orchestra.run.smithi076.stdout:49 read 0xc920293 thru 0xc920cfd (0xa6b bytes) 2024-09-16T13:28:31.222 INFO:teuthology.orchestra.run.smithi076.stdout:50 punch from 0x8e7525d to 0x8e82205, (0xcfa8 bytes) 2024-09-16T13:28:31.222 INFO:teuthology.orchestra.run.smithi076.stdout:51 read 0x32b9f16 thru 0x32c4ec4 (0xafaf bytes) 2024-09-16T13:28:31.223 INFO:teuthology.orchestra.run.smithi076.stdout:53 read 0xaa22f04 thru 0xaa2fda7 (0xcea4 bytes) 2024-09-16T13:28:31.223 INFO:teuthology.orchestra.run.smithi076.stdout:54 read 0x25288e8 thru 0x2530973 (0x808c bytes) 2024-09-16T13:28:31.223 INFO:teuthology.orchestra.run.smithi076.stdout:55 writesame 0x4d7532d thru 0x4d82da5 data_size 0x3(0xda79 bytes) 2024-09-16T13:28:31.237 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.238 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.242 INFO:teuthology.orchestra.run.smithi076.stdout:56 writesame 0x180ea90 thru 0x18107cb data_size 0x2(0x1d3c bytes) 2024-09-16T13:28:31.242 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:31.239+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:31.246 INFO:teuthology.orchestra.run.smithi076.stdout:57 punch from 0x5a2be68 to 0x5a38d6b, (0xcf03 bytes) 2024-09-16T13:28:31.246 INFO:teuthology.orchestra.run.smithi076.stdout:59 punch from 0xae528b3 to 0xae5daae, (0xb1fb bytes) 2024-09-16T13:28:31.246 INFO:teuthology.orchestra.run.smithi076.stdout:60 read 0xc520b90 thru 0xc527a00 (0x6e71 bytes) 2024-09-16T13:28:31.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:31.254 INFO:teuthology.orchestra.run.smithi076.stdout:61 write 0xe04b73e thru 0xe055bbb (0xa47e bytes) 2024-09-16T13:28:31.263 INFO:teuthology.orchestra.run.smithi076.stdout:62 trunc from 0xe055bbc to 0x9b6832d 2024-09-16T13:28:31.281 INFO:teuthology.orchestra.run.smithi076.stdout:63 write 0x1134b0d thru 0x113ea53 (0x9f47 bytes) 2024-09-16T13:28:31.292 INFO:teuthology.orchestra.run.smithi076.stdout:64 write 0xaeb688b thru 0xaec335d (0xcad3 bytes) 2024-09-16T13:28:31.295 INFO:teuthology.orchestra.run.smithi076.stdout:65 read 0x77ffdb9 thru 0x780bb27 (0xbd6f bytes) 2024-09-16T13:28:31.295 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.295 INFO:teuthology.orchestra.run.smithi076.stdout:68 punch from 0x7499451 to 0x74a737f, (0xdf2e bytes) 2024-09-16T13:28:31.295 INFO:teuthology.orchestra.run.smithi076.stdout:69 punch from 0x6958224 to 0x695f7b4, (0x7590 bytes) 2024-09-16T13:28:31.295 INFO:teuthology.orchestra.run.smithi076.stdout:70 writesame 0x3c35a51 thru 0x3c39d88 data_size 0x24(0x4338 bytes) 2024-09-16T13:28:31.298 INFO:teuthology.orchestra.run.smithi076.stdout:71 trunc from 0xaec335e to 0xb0ae308 2024-09-16T13:28:31.303 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.313 INFO:teuthology.orchestra.run.smithi076.stdout:73 writesame 0xdd72c09 thru 0xdd79c03 data_size 0x1(0x6ffb bytes) 2024-09-16T13:28:31.313 INFO:teuthology.orchestra.run.smithi076.stdout:74 writesame 0x1c8366c thru 0x1c8db51 data_size 0x1(0xa4e6 bytes) 2024-09-16T13:28:31.343 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:31.341+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:31.390 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.390 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.392 INFO:teuthology.orchestra.run.smithi076.stdout:75 read 0x2e30c3e thru 0x2e3a47c (0x983f bytes) 2024-09-16T13:28:31.392 INFO:teuthology.orchestra.run.smithi076.stdout:76 writesame 0x1b25f16 thru 0x1b2d720 data_size 0x1(0x780b bytes) 2024-09-16T13:28:31.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:31.445 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:31.442+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:31.449 INFO:teuthology.orchestra.run.smithi076.stdout:77 punch from 0x5959043 to 0x595e894, (0x5851 bytes) 2024-09-16T13:28:31.449 INFO:teuthology.orchestra.run.smithi076.stdout:78 read 0xc1f4df thru 0xc2756b (0x808d bytes) 2024-09-16T13:28:31.450 INFO:teuthology.orchestra.run.smithi076.stdout:79 punch from 0x9c625aa to 0x9c63893, (0x12e9 bytes) 2024-09-16T13:28:31.456 INFO:teuthology.orchestra.run.smithi076.stdout:80 write 0xe4fce5d thru 0xe4fd54a (0x6ee bytes) 2024-09-16T13:28:31.458 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.459 INFO:teuthology.orchestra.run.smithi076.stdout:82 write 0x31daa7 thru 0x3210e2 (0x363c bytes) 2024-09-16T13:28:31.462 INFO:teuthology.orchestra.run.smithi076.stdout:83 read 0x7b7beb5 thru 0x7b7edfd (0x2f49 bytes) 2024-09-16T13:28:31.462 INFO:teuthology.orchestra.run.smithi076.stdout:84 read 0x4672a6c thru 0x46790f1 (0x6686 bytes) 2024-09-16T13:28:31.462 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:86 writesame 0x9c8d5e0 thru 0x9c9125a data_size 0xd(0x3c7b bytes) 2024-09-16T13:28:31.465 INFO:teuthology.orchestra.run.smithi076.stdout:87 read 0x8e4194d thru 0x8e49367 (0x7a1b bytes) 2024-09-16T13:28:31.465 INFO:teuthology.orchestra.run.smithi076.stdout:89 punch from 0x2e1ae2 to 0x2e42de, (0x27fc bytes) 2024-09-16T13:28:31.465 INFO:teuthology.orchestra.run.smithi076.stdout:90 read 0x82cb603 thru 0x82d8527 (0xcf25 bytes) 2024-09-16T13:28:31.479 INFO:teuthology.orchestra.run.smithi076.stdout:91 trunc from 0xe4fd54b to 0x228daf6 2024-09-16T13:28:31.496 INFO:teuthology.orchestra.run.smithi076.stdout:92 punch from 0x14a511e to 0x14a93c1, (0x42a3 bytes) 2024-09-16T13:28:31.502 INFO:teuthology.orchestra.run.smithi076.stdout:93 writesame 0x2ab45c4 thru 0x2ac11ee data_size 0x1(0xcc2b bytes) 2024-09-16T13:28:31.502 INFO:teuthology.orchestra.run.smithi076.stdout:95 read 0x148b87c thru 0x149a35d (0xeae2 bytes) 2024-09-16T13:28:31.505 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.505 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.510 INFO:teuthology.orchestra.run.smithi076.stdout:96 trunc from 0x2ac11ef to 0x9748406 2024-09-16T13:28:31.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:31.515 INFO:teuthology.orchestra.run.smithi076.stdout:97 read 0x256bc2b thru 0x257a099 (0xe46f bytes) 2024-09-16T13:28:31.525 INFO:teuthology.orchestra.run.smithi076.stdout:99 write 0xd1c6d1c thru 0xd1cac01 (0x3ee6 bytes) 2024-09-16T13:28:31.528 INFO:teuthology.orchestra.run.smithi076.stdout:100 read 0x9315fd6 thru 0x931be56 (0x5e81 bytes) 2024-09-16T13:28:31.539 INFO:teuthology.orchestra.run.smithi076.stdout:101 trunc from 0xd1cac02 to 0x32f6e92 2024-09-16T13:28:31.549 INFO:teuthology.orchestra.run.smithi076.stdout:102 read 0x2b130ce thru 0x2b22d22 (0xfc55 bytes) 2024-09-16T13:28:31.549 INFO:teuthology.orchestra.run.smithi076.stdout:103 punch from 0x1150adf to 0x11544a5, (0x39c6 bytes) 2024-09-16T13:28:31.552 INFO:teuthology.orchestra.run.smithi076.stdout:104 trunc from 0x32f6e92 to 0x5b139db 2024-09-16T13:28:31.555 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:31.552+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:31.557 INFO:teuthology.orchestra.run.smithi076.stdout:105 punch from 0x414c8a3 to 0x414cd62, (0x4bf bytes) 2024-09-16T13:28:31.557 INFO:teuthology.orchestra.run.smithi076.stdout:107 read 0x33abb24 thru 0x33ae797 (0x2c74 bytes) 2024-09-16T13:28:31.557 INFO:teuthology.orchestra.run.smithi076.stdout:109 punch from 0x22d496e to 0x22d7d0c, (0x339e bytes) 2024-09-16T13:28:31.557 INFO:teuthology.orchestra.run.smithi076.stdout:110 read 0x3277118 thru 0x327d36e (0x6257 bytes) 2024-09-16T13:28:31.557 INFO:teuthology.orchestra.run.smithi076.stdout:111 writesame 0x237c59c thru 0x23884a0 data_size 0x1(0xbf05 bytes) 2024-09-16T13:28:31.615 INFO:teuthology.orchestra.run.smithi076.stdout:112 trunc from 0x5b139db to 0x443a093 2024-09-16T13:28:31.622 INFO:teuthology.orchestra.run.smithi076.stdout:114 punch from 0x1a26ef4 to 0x1a32e44, (0xbf50 bytes) 2024-09-16T13:28:31.629 INFO:teuthology.orchestra.run.smithi076.stdout:115 writesame 0x64dfac8 thru 0x64e88e6 data_size 0x1(0x8e1f bytes) 2024-09-16T13:28:31.629 INFO:teuthology.orchestra.run.smithi076.stdout:118 write 0x1b6b086 thru 0x1b76085 (0xb000 bytes) 2024-09-16T13:28:31.633 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.633 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:31.643 INFO:teuthology.orchestra.run.smithi076.stdout:123 writesame 0xa56553c thru 0xa568256 data_size 0x3(0x2d1b bytes) 2024-09-16T13:28:31.643 INFO:teuthology.orchestra.run.smithi076.stdout:124 writesame 0x8997b01 thru 0x89a3e16 data_size 0x1(0xc316 bytes) 2024-09-16T13:28:31.645 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:31.644+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:31.657 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.657 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:31.689 INFO:teuthology.orchestra.run.smithi076.stdout:125 writesame 0x86b4b3d thru 0x86bd881 data_size 0xf(0x8d45 bytes) 2024-09-16T13:28:31.694 INFO:teuthology.orchestra.run.smithi076.stdout:127 read 0x8df2ff9 thru 0x8dfbd78 (0x8d80 bytes) 2024-09-16T13:28:31.702 INFO:teuthology.orchestra.run.smithi076.stdout:129 write 0xc0be464 thru 0xc0ca19d (0xbd3a bytes) 2024-09-16T13:28:31.705 INFO:teuthology.orchestra.run.smithi076.stdout:130 punch from 0x895b7e8 to 0x895edcf, (0x35e7 bytes) 2024-09-16T13:28:31.705 INFO:teuthology.orchestra.run.smithi076.stdout:131 write 0x2e0b087 thru 0x2e170d0 (0xc04a bytes) 2024-09-16T13:28:31.710 INFO:teuthology.orchestra.run.smithi076.stdout:133 clone 1 order 22 su 131072 sc 12 2024-09-16T13:28:31.746 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:31.744+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:31.766 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.766 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:31.859 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:31.856+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:31.914 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:31.914 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:31.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:31.967 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:31.965+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:31.997 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:28:32.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:32.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:28:32.001 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T13:28:32.001 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.676666 2024-09-16T13:28:32.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.676665883602083 2024-09-16T13:28:32.037 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:32.037 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:32.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:32.049 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:32.046+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:32.150 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:32.147+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:32.173 INFO:teuthology.orchestra.run.smithi076.stdout:135 trunc from 0xc0ca19e to 0xbd1d9a8 2024-09-16T13:28:32.194 INFO:teuthology.orchestra.run.smithi076.stdout:137 punch from 0xa7e57e7 to 0xa7ea4f7, (0x4d10 bytes) 2024-09-16T13:28:32.194 INFO:teuthology.orchestra.run.smithi076.stdout:138 read 0xad4fb04 thru 0xad5c562 (0xca5f bytes) 2024-09-16T13:28:32.194 INFO:teuthology.orchestra.run.smithi076.stdout:139 write 0x79e792d thru 0x79e8762 (0xe36 bytes) 2024-09-16T13:28:32.198 INFO:teuthology.orchestra.run.smithi076.stdout:140 read 0x47bb236 thru 0x47bd0f0 (0x1ebb bytes) 2024-09-16T13:28:32.198 INFO:teuthology.orchestra.run.smithi076.stdout:141 read 0xe35196 thru 0xe41cbd (0xcb28 bytes) 2024-09-16T13:28:32.198 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:32.198 INFO:teuthology.orchestra.run.smithi076.stdout:143 write 0x1c58781 thru 0x1c5ccd4 (0x4554 bytes) 2024-09-16T13:28:32.200 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:32.200 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:32.202 INFO:teuthology.orchestra.run.smithi076.stdout:146 write 0xaf121e thru 0xb01101 (0xfee4 bytes) 2024-09-16T13:28:32.210 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:32.214 INFO:teuthology.orchestra.run.smithi076.stdout:148 write 0x15cff35 thru 0x15d8453 (0x851f bytes) 2024-09-16T13:28:32.219 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:32.220 INFO:teuthology.orchestra.run.smithi076.stdout:153 write 0x21127a2 thru 0x211ca7d (0xa2dc bytes) 2024-09-16T13:28:32.227 INFO:teuthology.orchestra.run.smithi076.stdout:154 read 0x3450dd2 thru 0x345fe15 (0xf044 bytes) 2024-09-16T13:28:32.227 INFO:teuthology.orchestra.run.smithi076.stdout:155 punch from 0x819c59d to 0x81a978f, (0xd1f2 bytes) 2024-09-16T13:28:32.227 INFO:teuthology.orchestra.run.smithi076.stdout:156 punch from 0x8eef797 to 0x8ef66dc, (0x6f45 bytes) 2024-09-16T13:28:32.227 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:32.227 INFO:teuthology.orchestra.run.smithi076.stdout:160 writesame 0x6fbac0f thru 0x6fc436e data_size 0x4(0x9760 bytes) 2024-09-16T13:28:32.234 INFO:teuthology.orchestra.run.smithi076.stdout:162 read 0x289181b thru 0x2894196 (0x297c bytes) 2024-09-16T13:28:32.242 INFO:teuthology.orchestra.run.smithi076.stdout:163 write 0xcf7f3f5 thru 0xcf814b0 (0x20bc bytes) 2024-09-16T13:28:32.249 INFO:teuthology.orchestra.run.smithi076.stdout:164 writesame 0x1eb685f thru 0x1ec2964 data_size 0x2(0xc106 bytes) 2024-09-16T13:28:32.250 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:32.247+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:32.264 INFO:teuthology.orchestra.run.smithi076.stdout:165 read 0x813214a thru 0x813251d (0x3d4 bytes) 2024-09-16T13:28:32.264 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xdb95d99 2024-09-16T13:28:32.265 INFO:teuthology.orchestra.run.smithi076.stdout:167 trunc from 0xcf814b1 to 0xdb95d99 2024-09-16T13:28:32.271 INFO:teuthology.orchestra.run.smithi076.stdout:170 punch from 0xa4b290f to 0xa4b71b1, (0x48a2 bytes) 2024-09-16T13:28:32.271 INFO:teuthology.orchestra.run.smithi076.stdout:173 write 0x2923490 thru 0x2924161 (0xcd2 bytes) 2024-09-16T13:28:32.272 INFO:teuthology.orchestra.run.smithi076.stdout:174 write 0x7297b48 thru 0x7298a96 (0xf4f bytes) 2024-09-16T13:28:32.275 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:32.280 INFO:teuthology.orchestra.run.smithi076.stdout:177 write 0xe0bee3d thru 0xe0c13ef (0x25b3 bytes) 2024-09-16T13:28:32.288 INFO:teuthology.orchestra.run.smithi076.stdout:179 write 0x925b455 thru 0x9261e90 (0x6a3c bytes) 2024-09-16T13:28:32.290 INFO:teuthology.orchestra.run.smithi076.stdout:180 read 0xd73967d thru 0xd744ef4 (0xb878 bytes) 2024-09-16T13:28:32.290 INFO:teuthology.orchestra.run.smithi076.stdout:181 write 0x69ba48c thru 0x69c5b27 (0xb69c bytes) 2024-09-16T13:28:32.300 INFO:teuthology.orchestra.run.smithi076.stdout:182 trunc from 0xe0c13f0 to 0x7bc1c52 2024-09-16T13:28:32.328 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:32.328 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:32.338 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:32.340 INFO:teuthology.orchestra.run.smithi076.stdout:184 write 0x897fd7b thru 0x8981e86 (0x210c bytes) 2024-09-16T13:28:32.343 INFO:teuthology.orchestra.run.smithi076.stdout:185 writesame 0x20d6b7d thru 0x20e48cd data_size 0x1(0xdd51 bytes) 2024-09-16T13:28:32.350 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:32.348+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:32.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:32.413 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:32.414 INFO:teuthology.orchestra.run.smithi076.stdout:188 read 0x57428aa thru 0x57523eb (0xfb42 bytes) 2024-09-16T13:28:32.414 INFO:teuthology.orchestra.run.smithi076.stdout:189 read 0x6b0b8da thru 0x6b1b4a3 (0xfbca bytes) 2024-09-16T13:28:32.414 INFO:teuthology.orchestra.run.smithi076.stdout:192 punch from 0x6a57c94 to 0x6a58911, (0xc7d bytes) 2024-09-16T13:28:32.414 INFO:teuthology.orchestra.run.smithi076.stdout:195 read 0x4757c08 thru 0x47602a2 (0x869b bytes) 2024-09-16T13:28:32.428 INFO:teuthology.orchestra.run.smithi076.stdout:196 writesame 0xeb3969b thru 0xeb4674f data_size 0x17(0xd0b5 bytes) 2024-09-16T13:28:32.428 INFO:teuthology.orchestra.run.smithi076.stdout:197 read 0xd31df13 thru 0xd31ff55 (0x2043 bytes) 2024-09-16T13:28:32.428 INFO:teuthology.orchestra.run.smithi076.stdout:198 write 0x9da76e5 thru 0x9dabdf1 (0x470d bytes) 2024-09-16T13:28:32.431 INFO:teuthology.orchestra.run.smithi076.stdout:199 writesame 0x8718a7a thru 0x871e898 data_size 0x3d(0x5e1f bytes) 2024-09-16T13:28:32.433 INFO:teuthology.orchestra.run.smithi076.stdout:200 read 0xc9c7881 thru 0xc9d3d5a (0xc4da bytes) 2024-09-16T13:28:32.433 INFO:teuthology.orchestra.run.smithi076.stdout:201 write 0x74c75ef thru 0x74ccdc2 (0x57d4 bytes) 2024-09-16T13:28:32.435 INFO:teuthology.orchestra.run.smithi076.stdout:202 writesame 0x940d918 thru 0x9418eb3 data_size 0x76(0xb59c bytes) 2024-09-16T13:28:32.443 INFO:teuthology.orchestra.run.smithi076.stdout:203 write 0x8c05585 thru 0x8c0c2ff (0x6d7b bytes) 2024-09-16T13:28:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:204 punch from 0x607309c to 0x607b038, (0x7f9c bytes) 2024-09-16T13:28:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:205 read 0xf82141 thru 0xf84d37 (0x2bf7 bytes) 2024-09-16T13:28:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:206 writesame 0x7bf0c7f thru 0x7bf8d8a data_size 0x6(0x810c bytes) 2024-09-16T13:28:32.450 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:32.450 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:32.451 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:32.449+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:32.456 INFO:teuthology.orchestra.run.smithi076.stdout:207 writesame 0xd7dfe26 thru 0xd7e2e3f data_size 0x2f(0x301a bytes) 2024-09-16T13:28:32.460 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:32.463 INFO:teuthology.orchestra.run.smithi076.stdout:210 write 0x9bd31cb thru 0x9bdd715 (0xa54b bytes) 2024-09-16T13:28:32.467 INFO:teuthology.orchestra.run.smithi076.stdout:211 write 0x8d9949a thru 0x8da3f93 (0xaafa bytes) 2024-09-16T13:28:32.469 INFO:teuthology.orchestra.run.smithi076.stdout:212 punch from 0x213dd32 to 0x213f067, (0x1335 bytes) 2024-09-16T13:28:32.469 INFO:teuthology.orchestra.run.smithi076.stdout:215 write 0xcec6672 thru 0xced4484 (0xde13 bytes) 2024-09-16T13:28:32.474 INFO:teuthology.orchestra.run.smithi076.stdout:216 clone 2 order 19 su 32768 sc 8 2024-09-16T13:28:32.495 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:32.510 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:32.526 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:32.542 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:32.552 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:32.550+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:32.557 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:32.557 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:32.563 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:32.563 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:32.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:32.573 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:32.573 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:32.588 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:32.589 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:32.604 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:32.604 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:32.653 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:32.651+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:5 2.2 e21/19 2:5cea7035:::rbd_info:head v 21'1, mlcod=0'0)", 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.362282654000001, 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011418870000000001, 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308443+0000", 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308444+0000", 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308452+0000", 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7539999999999995e-06 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308482+0000", 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9589000000000001e-05 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308497+0000", 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4671999999999999e-05 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308579+0000", 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2148e-05 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.308608+0000", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9612e-05 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.309541+0000", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00093304800000000002 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:11.309585+0000", 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3538000000000002e-05 2024-09-16T13:28:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:7 2.2 e22/19 2:5cea7035:::rbd_info:head v 22'2, mlcod=0'0)", 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.353226866, 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00083179599999999999, 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317499+0000", 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.16e-07 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317504+0000", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7539999999999997e-06 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317518+0000", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4516e-05 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317527+0000", 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0180000000000007e-06 2024-09-16T13:28:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317583+0000", 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5659999999999999e-05 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.317619+0000", 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6365000000000002e-05 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.318296+0000", 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067628100000000004 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:12.318331+0000", 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4986e-05 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:10 4.1c e35/32 4:3831950c:::rbd_directory:head v 35'1, mlcod=0'0)", 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.6847058320000001, 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054675700000000004, 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986020+0000", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986025+0000", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3020000000000002e-06 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986041+0000", 2024-09-16T13:28:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5993000000000001e-05 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986047+0000", 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986086+0000", 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8921999999999998e-05 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986105+0000", 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9364999999999998e-05 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986537+0000", 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043148900000000002 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986567+0000", 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9963000000000001e-05 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:32.686 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:32.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:32.753 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:32.752+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:32.784 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:32.784 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:32.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:32.854 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:32.852+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:32.930 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:32.930 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:32.930 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:32.930 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:32.930 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:32.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:32.983 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:32.981+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:32.998 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.2 primary-affinity to 0.676666 (8443452) 2024-09-16T13:28:33.043 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:33.043 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:33.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:33.043 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:33.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4257.0:1 2.3 e20/19 2:c4c92e5a:::rbd_trash:head v 20'1, mlcod=0'0)", 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.729527207, 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045403689999999998, 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312620+0000", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312624+0000", 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5340000000000001e-06 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312635+0000", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1029000000000001e-05 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312643+0000", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0290000000000005e-06 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312677+0000", 2024-09-16T13:28:33.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3207000000000002e-05 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.312699+0000", 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2226e-05 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.317133+0000", 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044345080000000002 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:10.317160+0000", 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6548999999999999e-05 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:10 4.1c 4:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.05683945, 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014440410000000001, 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.048 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 10 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985307+0000", 2024-09-16T13:28:33.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985309+0000", 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.885e-06 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985316+0000", 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7669999999999998e-06 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985329+0000", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2631e-05 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985337+0000", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8380000000000008e-06 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985394+0000", 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6873000000000002e-05 2024-09-16T13:28:33.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985539+0000", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144659 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.985753+0000", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021416 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986407+0000", 2024-09-16T13:28:33.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00065385200000000001 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986663+0000", 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025650800000000002 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986711+0000", 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7620999999999997e-05 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986751+0000", 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0247000000000003e-05 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:10 4.1c e35/32 ondisk, result = 0)", 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.055549675, 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016042399999999999, 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986597+0000", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986598+0000", 2024-09-16T13:28:33.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986602+0000", 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986611+0000", 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4760000000000004e-06 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986616+0000", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3320000000000002e-06 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986649+0000", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3083000000000002e-05 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986658+0000", 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.234e-06 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.986757+0000", 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9628000000000002e-05 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.058 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:33.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:33.060 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone1 from 0xeb46750 (overlap 0x7bc1c52) to 0x6f6197d 2024-09-16T13:28:33.060 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:33.057+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:33.161 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:33.158+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:33.172 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-09-16T13:28:33.207 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:33.207 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:33.207 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:33.208 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:33.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:33.261 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:33.259+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:33.323 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:33.324 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:33.324 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:33.324 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:33.324 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:33.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:33.377 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:33.375+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:33.395 INFO:teuthology.orchestra.run.smithi076.stdout:218 trunc from 0xeb46750 to 0x900ece1 2024-09-16T13:28:33.412 INFO:teuthology.orchestra.run.smithi076.stdout:221 punch from 0x8d24863 to 0x8d25d1b, (0x14b8 bytes) 2024-09-16T13:28:33.415 INFO:teuthology.orchestra.run.smithi076.stdout:222 trunc from 0x900ece1 to 0xbbc5407 2024-09-16T13:28:33.430 INFO:teuthology.orchestra.run.smithi076.stdout:224 writesame 0xee294bd thru 0xee32f13 data_size 0x1(0x9a57 bytes) 2024-09-16T13:28:33.430 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:33.430 INFO:teuthology.orchestra.run.smithi076.stdout:226 write 0x512e9f8 thru 0x5135ce0 (0x72e9 bytes) 2024-09-16T13:28:33.436 INFO:teuthology.orchestra.run.smithi076.stdout:227 read 0x898407 thru 0x89c484 (0x407e bytes) 2024-09-16T13:28:33.437 INFO:teuthology.orchestra.run.smithi076.stdout:228 read 0xbff9393 thru 0xbffc62c (0x329a bytes) 2024-09-16T13:28:33.437 INFO:teuthology.orchestra.run.smithi076.stdout:229 write 0xcc5b359 thru 0xcc61499 (0x6141 bytes) 2024-09-16T13:28:33.441 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:33.441 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:33.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:33.441 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:9 4.1a e35/32 4:5cea7035:::rbd_info:head v 35'2, mlcod=0'0)", 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.981676+0000", 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4566623219999997, 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023225530000000002, 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981676+0000", 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981676+0000", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981677+0000", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-07 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981684+0000", 2024-09-16T13:28:33.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4189999999999999e-06 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981694+0000", 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0266e-05 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981702+0000", 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9589999999999995e-06 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981783+0000", 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1668999999999993e-05 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.981806+0000", 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.285e-05 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.983963+0000", 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00215701 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.983999+0000", 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5753999999999997e-05 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:13 4.11 4:8f19f749:::rbd_object_map.11a9bbb7ef8c:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:28.993254+0000", 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4450850429999997, 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001537617, 2024-09-16T13:28:33.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 13 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993254+0000", 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993254+0000", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993256+0000", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.092e-06 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993261+0000", 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8679999999999998e-06 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993270+0000", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2900000000000008e-06 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993276+0000", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9789999999999998e-06 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993315+0000", 2024-09-16T13:28:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9062e-05 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993421+0000", 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010616300000000001 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.993545+0000", 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012386 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.994198+0000", 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00065273700000000002 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.994724+0000", 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000525725 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.994754+0000", 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0871999999999999e-05 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:28.994791+0000", 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6968999999999998e-05 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:856 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [read 0~49] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:32.999790+0000", 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.43854901200000002, 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021673080000000002, 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 856 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999790+0000", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999790+0000", 2024-09-16T13:28:33.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999792+0000", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1220000000000002e-06 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999801+0000", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8270000000000004e-06 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999822+0000", 2024-09-16T13:28:33.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0942999999999999e-05 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999830+0000", 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999877+0000", 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6192000000000001e-05 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:32.999881+0000", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5940000000000004e-06 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.001729+0000", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001847418 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.001831+0000", 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102565 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.001957+0000", 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125998 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:858 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.005393+0000", 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.43294542899999999, 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014052450000000001, 2024-09-16T13:28:33.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 858 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005393+0000", 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005393+0000", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005395+0000", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.807e-06 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005404+0000", 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3960000000000006e-06 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005420+0000", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6143000000000001e-05 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005426+0000", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8119999999999996e-06 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005482+0000", 2024-09-16T13:28:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199000000000002e-05 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005582+0000", 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100691 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.005865+0000", 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028274300000000001 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.006580+0000", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071511899999999995 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.006703+0000", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012276 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.006753+0000", 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9515000000000003e-05 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.006799+0000", 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6060000000000003e-05 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:937 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.057559+0000", 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.38077938900000002, 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017410119999999999, 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 937 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057559+0000", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057559+0000", 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057561+0000", 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.68e-06 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057568+0000", 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057584+0000", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6042999999999999e-05 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057590+0000", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057640+0000", 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9842999999999997e-05 2024-09-16T13:28:33.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.057731+0000", 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1429999999999997e-05 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.058116+0000", 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038539599999999998 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.058883+0000", 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00076678499999999997 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059246+0000", 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000363066 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059278+0000", 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1985e-05 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059300+0000", 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2147e-05 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:938 4.19 4:9e284e47:::rbd_data.11a9c8c70c30.0000000000000025:head [delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.059390+0000", 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37894899399999998, 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021103129999999999, 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.469 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 938 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059390+0000", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059390+0000", 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059391+0000", 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.850000000000001e-07 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059395+0000", 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5879999999999999e-06 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059408+0000", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2588999999999999e-05 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059414+0000", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.314e-06 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059446+0000", 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1525000000000001e-05 2024-09-16T13:28:33.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059501+0000", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4796999999999997e-05 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059705+0000", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000204155 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.060892+0000", 2024-09-16T13:28:33.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001187085 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061438+0000", 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054583500000000005 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061474+0000", 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999997e-05 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061500+0000", 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6040000000000001e-05 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:940 4.1d 4:bf482af4:::rbd_data.11a9c8c70c30.000000000000002e:head [delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.059434+0000", 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37890498099999997, 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021681199999999999, 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 940 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059434+0000", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059434+0000", 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059434+0000", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8800000000000002e-07 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059437+0000", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9019999999999999e-06 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059443+0000", 2024-09-16T13:28:33.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5480000000000001e-06 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059446+0000", 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7800000000000001e-06 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059486+0000", 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999000000000001e-05 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059564+0000", 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8175000000000003e-05 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059736+0000", 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000172237 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.060625+0000", 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00088864399999999996 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061536+0000", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00091129500000000001 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061567+0000", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1121000000000002e-05 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061602+0000", 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4830999999999999e-05 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:941 4.f e37/32 4:f081575a:::rbd_data.11a9c8c70c30.000000000000003a:head v 37'9, mlcod=36'6)", 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.059815+0000", 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.378523844, 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001657915, 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059815+0000", 2024-09-16T13:28:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059815+0000", 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059815+0000", 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2500000000000001e-07 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059822+0000", 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059831+0000", 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8740000000000001e-06 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059836+0000", 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3399999999999997e-06 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059877+0000", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1162000000000003e-05 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.059897+0000", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9344999999999999e-05 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061452+0000", 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001555666 2024-09-16T13:28:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.061473+0000", 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0682999999999998e-05 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:944 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065460+0000", 2024-09-16T13:28:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37287889899999999, 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022258080000000001, 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 944 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065460+0000", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065460+0000", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065460+0000", 2024-09-16T13:28:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7100000000000002e-07 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065464+0000", 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8130000000000002e-06 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065471+0000", 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1080000000000004e-06 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065476+0000", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7280000000000003e-06 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065504+0000", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8065e-05 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065548+0000", 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4104000000000003e-05 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065724+0000", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017531600000000001 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067589+0000", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018650559999999999 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067597+0000", 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4479999999999994e-06 2024-09-16T13:28:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067646+0000", 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8690000000000003e-05 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067686+0000", 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9909e-05 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:945 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065521+0000", 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37281733099999997, 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022770400000000001, 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 945 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065521+0000", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065521+0000", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065522+0000", 2024-09-16T13:28:33.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2500000000000005e-07 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065525+0000", 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2380000000000002e-06 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065533+0000", 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6520000000000006e-06 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065537+0000", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2590000000000002e-06 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065977+0000", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044004899999999997 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.066034+0000", 2024-09-16T13:28:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6227999999999998e-05 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.066236+0000", 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000202079 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067696+0000", 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014605009999999999 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067728+0000", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1572000000000002e-05 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067776+0000", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8523000000000003e-05 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067798+0000", 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2314e-05 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:946 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065544+0000", 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37279505000000002, 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051923999999999998, 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 946 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065544+0000", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065544+0000", 2024-09-16T13:28:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065544+0000", 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7899999999999999e-07 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065547+0000", 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065554+0000", 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3710000000000002e-06 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065557+0000", 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.066504+0000", 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094610899999999999 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.066550+0000", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6969999999999999e-05 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.066790+0000", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000239029 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.068378+0000", 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001588707 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070651+0000", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022722839999999998 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070698+0000", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7707999999999998e-05 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070736+0000", 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7917999999999998e-05 2024-09-16T13:28:33.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:947 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065561+0000", 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37277732800000002, 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077154470000000003, 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.504 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 947 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065561+0000", 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065561+0000", 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065562+0000", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0800000000000001e-07 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065564+0000", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1519999999999999e-06 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065570+0000", 2024-09-16T13:28:33.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7710000000000003e-06 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065573+0000", 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.602e-06 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067047+0000", 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014736319999999999 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067102+0000", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067321+0000", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021841100000000001 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069815+0000", 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002494254 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073221+0000", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003406156 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073256+0000", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4983e-05 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073277+0000", 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0948e-05 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.509 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:948 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065577+0000", 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37276169999999997, 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078116590000000003, 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 948 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.510 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065577+0000", 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065577+0000", 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065577+0000", 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7399999999999999e-07 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065581+0000", 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.095e-06 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065587+0000", 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.224e-06 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065590+0000", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.365e-06 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067821+0000", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022312120000000002 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.067888+0000", 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6851000000000001e-05 2024-09-16T13:28:33.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.068077+0000", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018831800000000001 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070750+0000", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026730859999999999 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073315+0000", 2024-09-16T13:28:33.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025657229999999998 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073354+0000", 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8285999999999999e-05 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073389+0000", 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5125e-05 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.515 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:949 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065594+0000", 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37274504600000002, 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078865970000000004, 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.516 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 949 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065594+0000", 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065594+0000", 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065594+0000", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065596+0000", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3269999999999999e-06 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065602+0000", 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065607+0000", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9139999999999999e-06 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.068426+0000", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028186449999999998 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.068519+0000", 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2966000000000004e-05 2024-09-16T13:28:33.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.068820+0000", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030158499999999999 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071943+0000", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031232500000000002 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073434+0000", 2024-09-16T13:28:33.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014906120000000001 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073460+0000", 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5624999999999999e-05 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073480+0000", 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0615000000000002e-05 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:950 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065612+0000", 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.372726738, 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079513880000000002, 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 950 2024-09-16T13:28:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065612+0000", 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065612+0000", 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065612+0000", 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9000000000000002e-07 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065616+0000", 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.179e-06 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065621+0000", 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065625+0000", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2650000000000001e-06 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069154+0000", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035294160000000001 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069232+0000", 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7888999999999999e-05 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.526 INFO:teuthology.orchestra.run.smithi076.stdout:230 writesame 0xd8840a6 thru 0xd88aa44 data_size 0x1(0x699f bytes) 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069503+0000", 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027085400000000001 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071947+0000", 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002444257 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073510+0000", 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015634819999999999 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073534+0000", 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3985000000000002e-05 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073563+0000", 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8975e-05 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:951 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065628+0000", 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37271041999999999, 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080688270000000006, 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.529 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 951 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065628+0000", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065628+0000", 2024-09-16T13:28:33.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065629+0000", 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065631+0000", 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9750000000000001e-06 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065637+0000", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8699999999999997e-06 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065641+0000", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2760000000000002e-06 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069856+0000", 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042154109999999996 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.069938+0000", 2024-09-16T13:28:33.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1267000000000006e-05 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070285+0000", 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000347115 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073189+0000", 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029040300000000002 2024-09-16T13:28:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073615+0000", 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042634500000000001 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073662+0000", 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6829999999999997e-05 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073697+0000", 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5330000000000002e-05 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:952 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065655+0000", 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37268378200000002, 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082272850000000008, 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 952 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065655+0000", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065655+0000", 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065655+0000", 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6899999999999998e-07 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065658+0000", 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6850000000000001e-06 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065665+0000", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.153e-06 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065669+0000", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070778+0000", 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051086600000000001 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.070826+0000", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8299e-05 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071025+0000", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000199051 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073401+0000", 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023753649999999999 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073788+0000", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738599999999998 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073838+0000", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0210000000000002e-05 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073882+0000", 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4129999999999999e-05 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:953 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065673+0000", 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.372665631, 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083246399999999995, 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 953 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065673+0000", 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065673+0000", 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065674+0000", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065676+0000", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065681+0000", 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8510000000000004e-06 2024-09-16T13:28:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065685+0000", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2940000000000001e-06 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071306+0000", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056213920000000002 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071398+0000", 2024-09-16T13:28:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2002000000000004e-05 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071722+0000", 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000323751 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073407+0000", 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001684748 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073928+0000", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000520884 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073967+0000", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.985e-05 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073998+0000", 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0391000000000001e-05 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:954 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065688+0000", 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.372650645, 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084263610000000003, 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 954 2024-09-16T13:28:33.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065688+0000", 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065688+0000", 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065688+0000", 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5100000000000001e-07 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065691+0000", 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.119e-06 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065696+0000", 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4330000000000003e-06 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065699+0000", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.269e-06 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.071975+0000", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062752889999999999 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.072038+0000", 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3634999999999996e-05 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.072263+0000", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022518399999999999 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073413+0000", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491800000000001 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074037+0000", 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062457099999999998 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074077+0000", 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9708e-05 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074115+0000", 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7622e-05 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.555 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:955 4.0 4:014568b2:::rbd_object_map.11a9c8c70c30:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.065703+0000", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.37263587399999998, 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085161279999999995, 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 955 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065703+0000", 2024-09-16T13:28:33.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065703+0000", 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065703+0000", 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:33.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065705+0000", 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0839999999999999e-06 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065711+0000", 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5809999999999996e-06 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.065714+0000", 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1480000000000002e-06 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.072566+0000", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068520539999999998 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.072649+0000", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2749000000000003e-05 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.072955+0000", 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000305896 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.073897+0000", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094195399999999995 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074149+0000", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025194699999999998 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074185+0000", 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6668000000000002e-05 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:33.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.074219+0000", 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3689000000000002e-05 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:33.561 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:33.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:231 read 0x7e028b8 thru 0x7e0ac9e (0x83e7 bytes) 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:232 write 0xe747f38 thru 0xe74f601 (0x76ca bytes) 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:234 writesame 0x8ae72d9 thru 0x8aed680 data_size 0xc(0x63a8 bytes) 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:235 trunc from 0xee32f14 to 0x349b9c6 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:236 write 0x348f93 thru 0x350223 (0x7291 bytes) 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:238 write 0xb337d25 thru 0xb33986c (0x1b48 bytes) 2024-09-16T13:28:33.563 INFO:teuthology.orchestra.run.smithi076.stdout:240 trunc from 0xb33986d to 0x2f25dde 2024-09-16T13:28:33.564 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:33.562+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:33.595 INFO:teuthology.orchestra.run.smithi076.stdout:241 read 0x225f021 thru 0x2263a96 (0x4a76 bytes) 2024-09-16T13:28:33.595 INFO:teuthology.orchestra.run.smithi076.stdout:242 write 0xa7adb thru 0xa7f85 (0x4ab bytes) 2024-09-16T13:28:33.608 INFO:teuthology.orchestra.run.smithi076.stdout:243 trunc from 0x2f25dde to 0x530e066 2024-09-16T13:28:33.617 INFO:teuthology.orchestra.run.smithi076.stdout:244 punch from 0x17e28ee to 0x17e5d11, (0x3423 bytes) 2024-09-16T13:28:33.617 INFO:teuthology.orchestra.run.smithi076.stdout:245 trunc from 0x530e066 to 0x562b70f 2024-09-16T13:28:33.621 INFO:teuthology.orchestra.run.smithi076.stdout:246 read 0x4105ee9 thru 0x410e904 (0x8a1c bytes) 2024-09-16T13:28:33.621 INFO:teuthology.orchestra.run.smithi076.stdout:247 write 0x4522941 thru 0x452d713 (0xadd3 bytes) 2024-09-16T13:28:33.627 INFO:teuthology.orchestra.run.smithi076.stdout:249 punch from 0x4b8a4ab to 0x4b93867, (0x93bc bytes) 2024-09-16T13:28:33.638 INFO:teuthology.orchestra.run.smithi076.stdout:250 write 0x951b61f thru 0x952398b (0x836d bytes) 2024-09-16T13:28:33.645 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:33.645 INFO:teuthology.orchestra.run.smithi076.stdout:252 writesame 0x5273cbe thru 0x527eddb data_size 0x63(0xb11e bytes) 2024-09-16T13:28:33.652 INFO:teuthology.orchestra.run.smithi076.stdout:254 read 0x901d8eb thru 0x9025b7b (0x8291 bytes) 2024-09-16T13:28:33.653 INFO:teuthology.orchestra.run.smithi076.stdout:255 trunc from 0x952398c to 0xa2c46ff 2024-09-16T13:28:33.658 INFO:teuthology.orchestra.run.smithi076.stdout:256 read 0x12372ad thru 0x12455c7 (0xe31b bytes) 2024-09-16T13:28:33.659 INFO:teuthology.orchestra.run.smithi076.stdout:257 writesame 0xc38178 thru 0xc3cb8b data_size 0x16(0x4a14 bytes) 2024-09-16T13:28:33.665 INFO:teuthology.orchestra.run.smithi076.stdout:258 read 0x610fff5 thru 0x6117743 (0x774f bytes) 2024-09-16T13:28:33.665 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:33.662+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:33.668 INFO:teuthology.orchestra.run.smithi076.stdout:259 trunc from 0xa2c46ff to 0xd2b04f6 2024-09-16T13:28:33.671 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1532 4.18 4:18a07cda:::rbd_header.11a962dad130:head [call rbd.set_size in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.669258+0000", 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.001284915, 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013031749999999999, 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1532 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669258+0000", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669258+0000", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669258+0000", 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4499999999999997e-07 2024-09-16T13:28:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669261+0000", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1779999999999999e-06 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669269+0000", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669273+0000", 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.591e-06 2024-09-16T13:28:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669303+0000", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0497e-05 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669353+0000", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9920999999999999e-05 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.669462+0000", 2024-09-16T13:28:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000109344 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:33.676 INFO:teuthology.orchestra.run.smithi076.stdout:260 write 0xc8db0f1 thru 0xc8e43d8 (0x92e8 bytes) 2024-09-16T13:28:33.681 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:33.682 INFO:teuthology.orchestra.run.smithi076.stdout:261 read 0x58ad498 thru 0x58b3a81 (0x65ea bytes) 2024-09-16T13:28:33.682 INFO:teuthology.orchestra.run.smithi076.stdout:262 writesame 0x50e7456 thru 0x50f230e data_size 0x1(0xaeb9 bytes) 2024-09-16T13:28:33.740 INFO:teuthology.orchestra.run.smithi076.stdout:263 write 0x8bac5fb thru 0x8baf746 (0x314c bytes) 2024-09-16T13:28:33.743 INFO:teuthology.orchestra.run.smithi076.stdout:264 write 0xaf562d1 thru 0xaf5e721 (0x8451 bytes) 2024-09-16T13:28:33.752 INFO:teuthology.orchestra.run.smithi076.stdout:265 trunc from 0xd2b04f6 to 0xac3f29e 2024-09-16T13:28:33.766 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:33.764+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:33.767 INFO:teuthology.orchestra.run.smithi076.stdout:266 writesame 0x7e6d2d0 thru 0x7e711c9 data_size 0x3(0x3efa bytes) 2024-09-16T13:28:33.773 INFO:teuthology.orchestra.run.smithi076.stdout:267 writesame 0x3b02aa9 thru 0x3b09cd8 data_size 0x7e(0x7230 bytes) 2024-09-16T13:28:33.777 INFO:teuthology.orchestra.run.smithi076.stdout:270 clone 3 order 24 su 524288 sc 15 2024-09-16T13:28:33.823 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:33.823 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:33.823 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:33.824 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:33.824 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:33.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:33.868 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:33.866+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:33.970 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:33.968+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:33.972 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1105 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.399696+0000", 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.56957093599999997, 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046931680000000002, 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1105 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399696+0000", 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399696+0000", 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399697+0000", 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8500000000000001e-07 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399700+0000", 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149999999999999e-06 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399706+0000", 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7869999999999998e-06 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.399709+0000", 2024-09-16T13:28:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1820000000000002e-06 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.400298+0000", 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058838800000000002 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.400344+0000", 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6480000000000002e-05 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.400536+0000", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000192091 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.401185+0000", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00064865400000000003 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.404341+0000", 2024-09-16T13:28:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031561580000000001 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.404369+0000", 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8265000000000001e-05 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.404389+0000", 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0163e-05 2024-09-16T13:28:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1204 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.503930+0000", 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46533731299999997, 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052627120000000001, 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1204 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503930+0000", 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503930+0000", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503931+0000", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1769999999999999e-06 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503936+0000", 2024-09-16T13:28:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.31e-06 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503950+0000", 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4173e-05 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.503980+0000", 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9315999999999998e-05 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504631+0000", 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00065074299999999996 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504688+0000", 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7364000000000002e-05 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504895+0000", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020701699999999999 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506143+0000", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012484880000000001 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509147+0000", 2024-09-16T13:28:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030033949999999998 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509172+0000", 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5384e-05 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509193+0000", 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0344999999999999e-05 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1205 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504025+0000", 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46524179100000002, 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052520500000000003, 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1205 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504025+0000", 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504025+0000", 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504027+0000", 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.353e-06 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504034+0000", 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5100000000000001e-06 2024-09-16T13:28:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504053+0000", 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9193000000000001e-05 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504064+0000", 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0723000000000001e-05 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505143+0000", 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010784740000000001 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505191+0000", 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8256e-05 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505420+0000", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000229464 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507217+0000", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001796714 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509228+0000", 2024-09-16T13:28:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020108460000000002 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509255+0000", 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7018000000000001e-05 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509277+0000", 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2498999999999999e-05 2024-09-16T13:28:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1206 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504087+0000", 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46518034899999999, 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054974300000000002, 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1206 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504087+0000", 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504087+0000", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504088+0000", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.277e-06 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504095+0000", 2024-09-16T13:28:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0060000000000003e-06 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504113+0000", 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7833e-05 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504123+0000", 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0446e-05 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505658+0000", 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001534723 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505703+0000", 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4938999999999998e-05 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.505927+0000", 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000223816 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507812+0000", 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001884707 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509546+0000", 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001734766 2024-09-16T13:28:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509566+0000", 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9578000000000001e-05 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509584+0000", 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8338999999999998e-05 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1207 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504140+0000", 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46512716700000001, 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055586519999999999, 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1207 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504140+0000", 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504140+0000", 2024-09-16T13:28:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504141+0000", 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504147+0000", 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.733e-06 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504176+0000", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8986e-05 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504179+0000", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.574e-06 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506174+0000", 2024-09-16T13:28:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019948990000000001 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506225+0000", 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0704000000000001e-05 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506462+0000", 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023658500000000001 2024-09-16T13:28:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.508298+0000", 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001836053 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509621+0000", 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001323662 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509665+0000", 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3399999999999998e-05 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509699+0000", 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3902000000000002e-05 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1208 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504187+0000", 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46507965499999998, 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0056127249999999998, 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1208 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504187+0000", 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504187+0000", 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504188+0000", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.75e-07 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504191+0000", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5660000000000001e-06 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504197+0000", 2024-09-16T13:28:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0859999999999998e-06 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504200+0000", 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3170000000000002e-06 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506705+0000", 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025055640000000001 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506754+0000", 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8142000000000001e-05 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.506953+0000", 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019933799999999999 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.508716+0000", 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001763148 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509753+0000", 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001037163 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509781+0000", 2024-09-16T13:28:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7484999999999999e-05 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509800+0000", 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9440999999999999e-05 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1209 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.014 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504216+0000", 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.46505097200000001, 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057177520000000004, 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1209 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504216+0000", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504216+0000", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504217+0000", 2024-09-16T13:28:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6100000000000001e-07 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504219+0000", 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7460000000000001e-06 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504227+0000", 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4909999999999999e-06 2024-09-16T13:28:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504231+0000", 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507336+0000", 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003105422 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507384+0000", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8168999999999999e-05 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507582+0000", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000197622 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509050+0000", 2024-09-16T13:28:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001467919 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509834+0000", 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00078408799999999995 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509898+0000", 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4555999999999997e-05 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509934+0000", 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5482999999999999e-05 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1210 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.504255+0000", 2024-09-16T13:28:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.465012023, 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058458390000000002, 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1210 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504255+0000", 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504255+0000", 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504256+0000", 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6300000000000005e-07 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504259+0000", 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2100000000000002e-06 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504266+0000", 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9959999999999996e-06 2024-09-16T13:28:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.504270+0000", 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2590000000000002e-06 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507831+0000", 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035607740000000001 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.507882+0000", 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1561000000000003e-05 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.508064+0000", 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018145600000000001 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.509079+0000", 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001015216 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.510017+0000", 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00093815000000000001 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.510065+0000", 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8180000000000003e-05 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.510101+0000", 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5474e-05 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1278 4.e 4:701e2443:::rbd_data.11a962dad130.0000000000000068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.028 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.560972+0000", 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.40829495399999999, 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046553430000000002, 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1278 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560972+0000", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560972+0000", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560973+0000", 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0800000000000004e-07 2024-09-16T13:28:34.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560976+0000", 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3289999999999998e-06 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560987+0000", 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.063e-05 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.560992+0000", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6909999999999996e-06 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561041+0000", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9104000000000003e-05 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561114+0000", 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3742e-05 2024-09-16T13:28:34.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561236+0000", 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012166400000000001 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.564718+0000", 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034817149999999998 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.565536+0000", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000818548 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.565591+0000", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4892999999999998e-05 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.565628+0000", 2024-09-16T13:28:34.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6319000000000002e-05 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1418 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.573309+0000", 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39595825600000001, 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068789929999999999, 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.035 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1418 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573309+0000", 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573309+0000", 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573310+0000", 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2299999999999999e-07 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573315+0000", 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7369999999999997e-06 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573329+0000", 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4012999999999999e-05 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573335+0000", 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.993e-06 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573685+0000", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035047799999999999 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.573765+0000", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0177000000000002e-05 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574131+0000", 2024-09-16T13:28:34.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036538299999999999 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575295+0000", 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011646440000000001 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.580093+0000", 2024-09-16T13:28:34.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047978200000000004 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.580144+0000", 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1023999999999998e-05 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.580188+0000", 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3801000000000003e-05 2024-09-16T13:28:34.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1419 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.574466+0000", 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39480134700000002, 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0087659390000000004, 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.042 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1419 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574466+0000", 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574466+0000", 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574467+0000", 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7000000000000003e-07 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574472+0000", 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.147e-06 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574483+0000", 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1201e-05 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574489+0000", 2024-09-16T13:28:34.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.519e-06 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574563+0000", 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4571000000000001e-05 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574647+0000", 2024-09-16T13:28:34.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3726000000000004e-05 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.574983+0000", 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033636000000000001 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575868+0000", 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00088438299999999998 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583166+0000", 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0072988159999999996 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583196+0000", 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9217000000000002e-05 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583232+0000", 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6028999999999997e-05 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1421 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.575905+0000", 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39336216099999999, 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073774239999999996, 2024-09-16T13:28:34.049 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1421 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575905+0000", 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575905+0000", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575906+0000", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4099999999999998e-07 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575911+0000", 2024-09-16T13:28:34.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8389999999999997e-06 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575923+0000", 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2221999999999999e-05 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575928+0000", 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5300000000000004e-06 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.575960+0000", 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1828000000000001e-05 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.576010+0000", 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9638999999999997e-05 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.576322+0000", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031264599999999997 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577136+0000", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081391799999999995 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583239+0000", 2024-09-16T13:28:34.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.006102405 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583262+0000", 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3328000000000001e-05 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583282+0000", 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0327999999999999e-05 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.055 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1422 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.577294+0000", 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.391973238, 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060326900000000003, 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1422 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577294+0000", 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577294+0000", 2024-09-16T13:28:34.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577295+0000", 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577299+0000", 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3839999999999999e-06 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577310+0000", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1253999999999999e-05 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577316+0000", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577356+0000", 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9984000000000002e-05 2024-09-16T13:28:34.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577438+0000", 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2224000000000004e-05 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.577800+0000", 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000361999 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.581261+0000", 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034607560000000002 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583287+0000", 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020263569999999999 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583307+0000", 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9474e-05 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583327+0000", 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9772e-05 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1423 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.579034+0000", 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39023299900000002, 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043388029999999996, 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1423 2024-09-16T13:28:34.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579034+0000", 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579034+0000", 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579035+0000", 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.61e-07 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579039+0000", 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0049999999999998e-06 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579049+0000", 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0283e-05 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579053+0000", 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9750000000000001e-06 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579091+0000", 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7756000000000002e-05 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579137+0000", 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6462999999999999e-05 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.579331+0000", 2024-09-16T13:28:34.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000193976 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.581569+0000", 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022380130000000001 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583336+0000", 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017667869999999999 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583357+0000", 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0665e-05 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.583373+0000", 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6118999999999999e-05 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1462 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.607585+0000", 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.361681898, 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041026129999999997, 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1462 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.070 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607585+0000", 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607585+0000", 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607586+0000", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607591+0000", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6489999999999999e-06 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607602+0000", 2024-09-16T13:28:34.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1082e-05 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607607+0000", 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4879999999999998e-06 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607654+0000", 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6470000000000001e-05 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.607746+0000", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1941999999999995e-05 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.608040+0000", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029374099999999998 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.608734+0000", 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00069424899999999997 2024-09-16T13:28:34.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.611593+0000", 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028592019999999999 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.611645+0000", 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1719999999999999e-05 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.611688+0000", 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3161000000000002e-05 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1481 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.621074+0000", 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.34819278999999997, 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039530600000000004, 2024-09-16T13:28:34.076 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1481 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621074+0000", 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621074+0000", 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621075+0000", 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2399999999999998e-07 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621078+0000", 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6560000000000001e-06 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621085+0000", 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2400000000000001e-06 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621089+0000", 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2100000000000003e-06 2024-09-16T13:28:34.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621624+0000", 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053477799999999997 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.621712+0000", 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7888999999999998e-05 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.622120+0000", 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040874100000000001 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.624674+0000", 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025539709999999999 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.624954+0000", 2024-09-16T13:28:34.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028013799999999998 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.624992+0000", 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7687e-05 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.625027+0000", 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5225999999999998e-05 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.082 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:1495 4.5 e37/32 4:a1e7256f:::rbd_data.11a962dad130.000000000000012b:head v 37'12, mlcod=37'10)", 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.639797+0000", 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.32946988399999999, 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043467080000000003, 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639797+0000", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639797+0000", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639798+0000", 2024-09-16T13:28:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7999999999999995e-07 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639814+0000", 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6452000000000001e-05 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639823+0000", 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5059999999999995e-06 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639826+0000", 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5590000000000001e-06 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639863+0000", 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-05 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.639891+0000", 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583999999999999e-05 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.644114+0000", 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042228309999999998 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.644144+0000", 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9776000000000001e-05 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.087 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1543 4.1 4:81788050:::rbd_data.11a962dad130.0000000000000194:head [set-alloc-hint object_size 524288 write_size 524288,write 98304~17369] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.677294+0000", 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.291972654, 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041992260000000003, 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1543 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677294+0000", 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677294+0000", 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677295+0000", 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1799999999999995e-07 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677314+0000", 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8597000000000002e-05 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677322+0000", 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1640000000000001e-06 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677326+0000", 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6140000000000002e-06 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677370+0000", 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3878999999999998e-05 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677457+0000", 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6599000000000002e-05 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677538+0000", 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1227e-05 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681413+0000", 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003875281 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681443+0000", 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9753000000000001e-05 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681472+0000", 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8637e-05 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681494+0000", 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1857000000000001e-05 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1545 4.c 4:3435ff18:::rbd_object_map.11a962dad130:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.681946+0000", 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.28732154799999998, 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049473879999999996, 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.095 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1545 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681946+0000", 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681946+0000", 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681946+0000", 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9999999999999999e-07 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681948+0000", 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0549999999999998e-06 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681952+0000", 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681954+0000", 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2000000000000001e-06 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.682502+0000", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00054750100000000004 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.682558+0000", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6638999999999998e-05 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.682764+0000", 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020544899999999999 2024-09-16T13:28:34.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683999+0000", 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00123562 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.686814+0000", 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028146109999999998 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.686858+0000", 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.384e-05 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.686893+0000", 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5308999999999997e-05 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1547 4.13 4:cd484016:::rbd_data.11a962dad130.00000000000000a4:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.684238+0000", 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.28502925499999998, 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052906969999999996, 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1547 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.102 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684238+0000", 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684238+0000", 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684239+0000", 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8199999999999999e-07 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684245+0000", 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2450000000000003e-06 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684253+0000", 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0620000000000001e-06 2024-09-16T13:28:34.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684257+0000", 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6729999999999998e-06 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684291+0000", 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4085000000000003e-05 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684370+0000", 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9487000000000007e-05 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.684690+0000", 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032024700000000001 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687007+0000", 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023161729999999999 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689465+0000", 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002458773 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689496+0000", 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0760999999999999e-05 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689529+0000", 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509000000000003e-05 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.108 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:34.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:28:34.110 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone2 from 0xac3f29e (overlap 0x2f25dde) to 0x96df92 2024-09-16T13:28:34.110 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:34.108+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:34.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:34.211 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:34.208+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:34.230 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:34.230 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:34.230 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:34.230 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:34.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:34.258 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-09-16T13:28:34.312 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:34.310+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:34.378 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:34.378 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:34.379 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:34.379 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:34.379 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:34.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:34.431 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:34.428+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:34.486 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:34.486 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:34.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:34.486 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4257.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:10.306358+0000", 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.177212235999999, 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011031226, 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4257", 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/289126807", 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306358+0000", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306358+0000", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306362+0000", 2024-09-16T13:28:34.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1989999999999999e-06 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306369+0000", 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0990000000000001e-06 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306384+0000", 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5053e-05 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306392+0000", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306461+0000", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8800000000000005e-05 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.306477+0000", 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6141999999999999e-05 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.312199+0000", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057220379999999996 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.312358+0000", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015926499999999999 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.312503+0000", 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014470100000000001 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.316380+0000", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038765430000000001 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.317325+0000", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00094534899999999995 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.317362+0000", 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7044e-05 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:10.317389+0000", 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6917e-05 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4257.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:11.304809+0000", 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.178761506000001, 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052977839999999998, 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.493 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4257", 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/289126807", 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304809+0000", 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304809+0000", 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304810+0000", 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2389999999999999e-06 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304819+0000", 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6770000000000007e-06 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304834+0000", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4212999999999999e-05 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304842+0000", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9610000000000005e-06 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304908+0000", 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5905e-05 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.304912+0000", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5040000000000004e-06 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.308034+0000", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031219889999999999 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.308137+0000", 2024-09-16T13:28:34.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102867 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.308252+0000", 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114679 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.308930+0000", 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00067877599999999997 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.309960+0000", 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010296299999999999 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.310041+0000", 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0995000000000006e-05 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:11.310106+0000", 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5349000000000001e-05 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4257.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:12.314043+0000", 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.169527145, 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004549928, 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4257", 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/289126807", 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314043+0000", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314043+0000", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314045+0000", 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3e-06 2024-09-16T13:28:34.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314053+0000", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.055e-06 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314071+0000", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7308000000000001e-05 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314080+0000", 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9660000000000002e-06 2024-09-16T13:28:34.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314140+0000", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0359000000000003e-05 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.314146+0000", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.317128+0000", 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029819439999999998 2024-09-16T13:28:34.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.317204+0000", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5811999999999999e-05 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.317312+0000", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108515 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.318138+0000", 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00082589100000000004 2024-09-16T13:28:34.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.318530+0000", 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039122899999999999 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.318565+0000", 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5574000000000002e-05 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:12.318593+0000", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7645999999999999e-05 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:939 4.8 4:14746996:::rbd_data.11a9c8c70c30.000000000000002c:head [delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.059456+0000", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.424114407, 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004667203, 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.506 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 939 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059456+0000", 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059456+0000", 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059457+0000", 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8619999999999999e-06 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059464+0000", 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8870000000000003e-06 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059481+0000", 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6663e-05 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059487+0000", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0900000000000001e-06 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059531+0000", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3714999999999997e-05 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.059535+0000", 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.515e-06 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.061170+0000", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001635051 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.061241+0000", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0287e-05 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.061340+0000", 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9340999999999996e-05 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.063956+0000", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026157229999999999 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.064035+0000", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9547000000000003e-05 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.064085+0000", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9651000000000003e-05 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.064123+0000", 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7870999999999997e-05 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:1284 4.7 e37/32 4:e5543b8d:::rbd_data.11a962dad130.000000000000006e:head v 37'6, mlcod=37'4)", 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.561467+0000", 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.92210328799999997, 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003121824, 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.512 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561467+0000", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561467+0000", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561468+0000", 2024-09-16T13:28:34.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5199999999999995e-07 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561474+0000", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3670000000000002e-06 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561485+0000", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0762e-05 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561492+0000", 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0960000000000003e-06 2024-09-16T13:28:34.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561536+0000", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4073000000000003e-05 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.561554+0000", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7791e-05 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.564546+0000", 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029920070000000001 2024-09-16T13:28:34.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.564589+0000", 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2775999999999998e-05 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:1543 4.1 e37/32 4:81788050:::rbd_data.11a962dad130.0000000000000194:head v 37'7, mlcod=35'5)", 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.677603+0000", 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.80596744399999998, 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003749301, 2024-09-16T13:28:34.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677603+0000", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677603+0000", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677604+0000", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.835e-06 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677625+0000", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0570000000000001e-05 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677639+0000", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4436e-05 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677645+0000", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.344e-06 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677689+0000", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3970000000000001e-05 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.677711+0000", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2024000000000001e-05 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681314+0000", 2024-09-16T13:28:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036030609999999999 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.681352+0000", 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8061e-05 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:1546 4.7 e37/32 4:e2e7f3c3:::rbd_data.11a962dad130.00000000000000a6:head v 37'9, mlcod=37'7)", 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.683344+0000", 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.80022633499999996, 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033142369999999998, 2024-09-16T13:28:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683344+0000", 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683344+0000", 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683345+0000", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.068e-06 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683591+0000", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024624400000000001 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683601+0000", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0223e-05 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683606+0000", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3599999999999998e-06 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683645+0000", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9127e-05 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.683668+0000", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2937000000000001e-05 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.686629+0000", 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029616909999999998 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.686658+0000", 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8586999999999999e-05 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1686 4.15 4:ae15a460:::rbd_data.11a962dad130.0000000000000021:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.064323+0000", 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.419247225, 2024-09-16T13:28:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036096219999999998, 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1686 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064323+0000", 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064323+0000", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064324+0000", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4339999999999999e-06 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064329+0000", 2024-09-16T13:28:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8600000000000001e-06 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064341+0000", 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.155e-05 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064346+0000", 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064387+0000", 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1013999999999997e-05 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.064391+0000", 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8279999999999996e-06 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.066535+0000", 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021441939999999999 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.066828+0000", 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029291199999999999 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.067932+0000", 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001104147 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:2013 4.8 4:115c302d:::rbd_data.11a962dad130.0000000000000026:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.084610+0000", 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39896028700000002, 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0071781479999999996, 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2013 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.084610+0000", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.084610+0000", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.084611+0000", 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0299999999999997e-07 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085441+0000", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00083029000000000002 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085455+0000", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3903000000000001e-05 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085461+0000", 2024-09-16T13:28:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5969999999999997e-06 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085497+0000", 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5753999999999997e-05 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085565+0000", 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8295999999999998e-05 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085760+0000", 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019437499999999999 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.091676+0000", 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005916569 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.091692+0000", 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5866e-05 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.091748+0000", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6156999999999998e-05 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.091788+0000", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9437999999999997e-05 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:2014 4.8 4:115c302d:::rbd_data.11a962dad130.0000000000000026:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.085473+0000", 2024-09-16T13:28:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39809715400000001, 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067544010000000002, 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2014 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085473+0000", 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085473+0000", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085473+0000", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1900000000000002e-07 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085476+0000", 2024-09-16T13:28:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085483+0000", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.085489+0000", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5679999999999999e-06 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.090311+0000", 2024-09-16T13:28:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048221790000000002 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.090384+0000", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3266e-05 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.090553+0000", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016927899999999999 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.091973+0000", 2024-09-16T13:28:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014200199999999999 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092160+0000", 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018687399999999999 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092192+0000", 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1452999999999999e-05 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092227+0000", 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5599000000000003e-05 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2025 4.1 e38/32 4:87ec5c63:::rbd_data.11a962dad130.0000000000000027:head v 38'11, mlcod=38'9)", 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.086790+0000", 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39677997199999998, 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054296429999999996, 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086790+0000", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086790+0000", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086791+0000", 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0809999999999999e-06 2024-09-16T13:28:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086937+0000", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014557799999999999 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086950+0000", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3458000000000001e-05 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086956+0000", 2024-09-16T13:28:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7289999999999997e-06 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.086998+0000", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2193000000000001e-05 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.087021+0000", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2467999999999999e-05 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092189+0000", 2024-09-16T13:28:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051688330000000003 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092220+0000", 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0303000000000001e-05 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:2070 4.8 4:17ec2bd4:::rbd_data.11a962dad130.0000000000000029:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.092805+0000", 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.39076468800000003, 2024-09-16T13:28:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052105930000000003, 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2070 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092805+0000", 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092805+0000", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.092807+0000", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3999999999999999e-06 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093338+0000", 2024-09-16T13:28:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053171900000000005 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093351+0000", 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2201999999999999e-05 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093357+0000", 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9789999999999998e-06 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093426+0000", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9308999999999997e-05 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093515+0000", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9467000000000002e-05 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093776+0000", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026103399999999999 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.097469+0000", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036926429999999998 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.097931+0000", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046191800000000002 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.097978+0000", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7046000000000003e-05 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.098016+0000", 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7876000000000001e-05 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:2071 4.8 4:17ec2bd4:::rbd_data.11a962dad130.0000000000000029:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.093369+0000", 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.390201346, 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049561880000000003, 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2071 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093369+0000", 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093369+0000", 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093369+0000", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093373+0000", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8380000000000004e-06 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093381+0000", 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7200000000000006e-06 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.093385+0000", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3460000000000004e-06 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.096674+0000", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003289092 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.096761+0000", 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6514999999999994e-05 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.096931+0000", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000170428 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.097791+0000", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00085976200000000003 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.098245+0000", 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045409299999999998 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.098288+0000", 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2327000000000003e-05 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.098325+0000", 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7286999999999997e-05 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:2711 4.15 4:a87bb541:::rbd_data.11a962dad130.0000000000000070:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.177111+0000", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.306459433, 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045416049999999998, 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2711 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177111+0000", 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177111+0000", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177112+0000", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8279999999999999e-06 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177117+0000", 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8060000000000002e-06 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177129+0000", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1609e-05 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177135+0000", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2539999999999998e-06 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177188+0000", 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3304000000000003e-05 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177270+0000", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1409000000000006e-05 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177435+0000", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016475699999999999 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.180989+0000", 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035548110000000002 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.181567+0000", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057744999999999997 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.181614+0000", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6955e-05 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.181652+0000", 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8421999999999999e-05 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2719 4.3 e38/32 4:c1bf8068:::rbd_data.11a962dad130.00000000000000fd:head v 38'11, mlcod=38'8)", 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.177972+0000", 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.305597858, 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0056692289999999996, 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177972+0000", 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177972+0000", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177973+0000", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177980+0000", 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2830000000000004e-06 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177991+0000", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1056999999999999e-05 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.177996+0000", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.938e-06 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178045+0000", 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8291000000000002e-05 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178066+0000", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1112999999999999e-05 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183604+0000", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055385369999999996 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183641+0000", 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7174e-05 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2717 4.7 e38/32 4:e2e7f3c3:::rbd_data.11a962dad130.00000000000000a6:head v 38'19, mlcod=38'16)", 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.178060+0000", 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.305510422, 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0056807050000000003, 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178060+0000", 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178060+0000", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178060+0000", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2099999999999995e-07 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178068+0000", 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8310000000000001e-06 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178079+0000", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1146e-05 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178085+0000", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9379999999999997e-06 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178232+0000", 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014693900000000001 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178253+0000", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1061999999999999e-05 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183709+0000", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054561460000000003 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183740+0000", 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0821999999999997e-05 2024-09-16T13:28:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2712 4.1f e38/32 4:fba26e5b:::rbd_data.11a962dad130.0000000000000071:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.178089+0000", 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.30548137400000003, 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0056185890000000002, 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178089+0000", 2024-09-16T13:28:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178089+0000", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178090+0000", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178106+0000", 2024-09-16T13:28:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6294e-05 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178118+0000", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1759999999999999e-05 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178123+0000", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8069999999999996e-06 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178258+0000", 2024-09-16T13:28:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013418499999999999 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178277+0000", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9653e-05 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183664+0000", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053866169999999998 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183707+0000", 2024-09-16T13:28:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3414999999999997e-05 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2718 4.1 e38/32 4:81e7b400:::rbd_data.11a962dad130.00000000000000a7:head v 38'18, mlcod=38'15)", 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.178104+0000", 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.30546577600000002, 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055582219999999998, 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178104+0000", 2024-09-16T13:28:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178104+0000", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178105+0000", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3000000000000001e-07 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178110+0000", 2024-09-16T13:28:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9259999999999999e-06 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178118+0000", 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5620000000000004e-06 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178123+0000", 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4259999999999996e-06 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178168+0000", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4817999999999997e-05 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.178187+0000", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9349000000000001e-05 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183628+0000", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054414179999999999 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.183662+0000", 2024-09-16T13:28:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4193000000000003e-05 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2821 4.1 e38/32 4:80d51c4d:::rbd_data.11a962dad130.0000000000000016:head v 38'19, mlcod=38'16)", 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.196245+0000", 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.287325149, 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0064192529999999998, 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196245+0000", 2024-09-16T13:28:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196245+0000", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196246+0000", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.204e-06 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196386+0000", 2024-09-16T13:28:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013985199999999999 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196397+0000", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1285e-05 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196405+0000", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5379999999999997e-06 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196449+0000", 2024-09-16T13:28:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4317999999999998e-05 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.196468+0000", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8672e-05 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.202626+0000", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.006158129 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.202664+0000", 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8254999999999999e-05 2024-09-16T13:28:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:2822 4.1 e38/32 4:80d51c4d:::rbd_data.11a962dad130.0000000000000016:head v 38'21, mlcod=38'16)", 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.199898+0000", 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.28367246400000001, 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030955219999999999, 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199898+0000", 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199898+0000", 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199898+0000", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1500000000000003e-07 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199905+0000", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9720000000000003e-06 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199916+0000", 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0740999999999999e-05 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199925+0000", 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.918e-06 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199972+0000", 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6814000000000003e-05 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.199991+0000", 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.906e-05 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.202966+0000", 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029751759999999999 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.202993+0000", 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7025999999999998e-05 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:34.587 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:34.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:34.588 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:34.586+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:34.599 INFO:teuthology.orchestra.run.smithi076.stdout:271 write 0x3daa97a thru 0x3daffcc (0x5653 bytes) 2024-09-16T13:28:34.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:34.633 INFO:teuthology.orchestra.run.smithi076.stdout:272 read 0x6cab144 thru 0x6cb8e51 (0xdd0e bytes) 2024-09-16T13:28:34.633 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:34.633 INFO:teuthology.orchestra.run.smithi076.stdout:276 punch from 0x8d68a3d to 0x8d765ad, (0xdb70 bytes) 2024-09-16T13:28:34.633 INFO:teuthology.orchestra.run.smithi076.stdout:277 write 0x27c1c24 thru 0x27d19f4 (0xfdd1 bytes) 2024-09-16T13:28:34.661 INFO:teuthology.orchestra.run.smithi076.stdout:279 write 0x17f9411 thru 0x17fbcdd (0x28cd bytes) 2024-09-16T13:28:34.688 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:34.687+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:34.698 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:34.699 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:34.699 INFO:teuthology.orchestra.run.smithi076.stdout:285 read 0x9d5a57c thru 0x9d68d91 (0xe816 bytes) 2024-09-16T13:28:34.699 INFO:teuthology.orchestra.run.smithi076.stdout:286 writesame 0x39e4e30 thru 0x39f2b91 data_size 0x2b(0xdd62 bytes) 2024-09-16T13:28:34.721 INFO:teuthology.orchestra.run.smithi076.stdout:287 writesame 0x25f5acc thru 0x2603a1c data_size 0x1(0xdf51 bytes) 2024-09-16T13:28:34.743 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:34.745 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:34.745 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:34.745 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:34.745 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:34.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:34.759 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:34.775 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:34.790 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:34.799 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:34.796+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:34.806 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:34.822 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:34.822 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:34.827 INFO:teuthology.orchestra.run.smithi076.stdout:288 writesame 0x9f814a5 thru 0x9f84f97 data_size 0x1(0x3af3 bytes) 2024-09-16T13:28:34.836 INFO:teuthology.orchestra.run.smithi076.stdout:289 read 0x470b021 thru 0x4719787 (0xe767 bytes) 2024-09-16T13:28:34.836 INFO:teuthology.orchestra.run.smithi076.stdout:290 read 0x94a2e48 thru 0x94ab66b (0x8824 bytes) 2024-09-16T13:28:34.836 INFO:teuthology.orchestra.run.smithi076.stdout:291 read 0x5d25cac thru 0x5d32757 (0xcaac bytes) 2024-09-16T13:28:34.836 INFO:teuthology.orchestra.run.smithi076.stdout:292 writesame 0x4d66566 thru 0x4d68fde data_size 0x1(0x2a79 bytes) 2024-09-16T13:28:34.840 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:34.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:34.841 INFO:teuthology.orchestra.run.smithi076.stdout:293 write 0x9e3af09 thru 0x9e40d9e (0x5e96 bytes) 2024-09-16T13:28:34.843 INFO:teuthology.orchestra.run.smithi076.stdout:294 punch from 0x22299f8 to 0x2238e4d, (0xf455 bytes) 2024-09-16T13:28:34.843 INFO:teuthology.orchestra.run.smithi076.stdout:295 clone 4 order 21 su 65536 sc 4 2024-09-16T13:28:34.856 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:34.856 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:34.857 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:34.857 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:34.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:34.907 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:34.904+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:34.910 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:28:34.985 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone3 from 0xac3f29e (overlap 0xac3f29e) to 0x0 2024-09-16T13:28:34.991 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:34.989+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:35.022 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.022 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:35.091 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:35.088+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:35.145 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-09-16T13:28:35.160 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.160 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:35.176 INFO:teuthology.orchestra.run.smithi076.stdout:297 trunc from 0xac3f29e to 0x3ab9b14 2024-09-16T13:28:35.191 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:35.190+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:298 write 0x911bac8 thru 0x912787b (0xbdb4 bytes) 2024-09-16T13:28:35.213 INFO:teuthology.orchestra.run.smithi076.stdout:299 read 0x55db369 thru 0x55de785 (0x341d bytes) 2024-09-16T13:28:35.213 INFO:teuthology.orchestra.run.smithi076.stdout:300 punch from 0x6b67180 to 0x6b718aa, (0xa72a bytes) 2024-09-16T13:28:35.213 INFO:teuthology.orchestra.run.smithi076.stdout:301 writesame 0x2f83a9f thru 0x2f8eedd data_size 0x9(0xb43f bytes) 2024-09-16T13:28:35.233 INFO:teuthology.orchestra.run.smithi076.stdout:302 writesame 0x3f9ad1a thru 0x3fa8084 data_size 0x1(0xd36b bytes) 2024-09-16T13:28:35.293 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:35.290+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:35.299 INFO:teuthology.orchestra.run.smithi076.stdout:303 clone 5 order 24 su 65536 sc 2 2024-09-16T13:28:35.312 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.313 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:35.406 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:35.403+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:35.459 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.459 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:35.509 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:35.507+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:35.567 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.567 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:35.621 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:35.619+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:35.673 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.673 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:35.723 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:35.721+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:35.797 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:35.795+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:35.820 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.820 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:35.898 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:35.897+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:35.927 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:35.927 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:35.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:36.000 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:35.998+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:36.039 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.039 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:36.093 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone4 from 0x912787c (overlap 0x3ab9b14) to 0x233c372 2024-09-16T13:28:36.100 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:36.098+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:36.169 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.169 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:36.221 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:36.219+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:36.238 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-09-16T13:28:36.239 INFO:teuthology.orchestra.run.smithi076.stdout:304 read 0x20ccd1 thru 0x21a6ed (0xda1d bytes) 2024-09-16T13:28:36.240 INFO:teuthology.orchestra.run.smithi076.stdout:305 read 0x30094d8 thru 0x300cb3f (0x3668 bytes) 2024-09-16T13:28:36.240 INFO:teuthology.orchestra.run.smithi076.stdout:306 write 0x1055f10 thru 0x105e305 (0x83f6 bytes) 2024-09-16T13:28:36.302 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:36.300+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:36.329 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.330 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.338 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:36.423 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:36.421+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:36.435 INFO:teuthology.orchestra.run.smithi076.stdout:307 trunc from 0x912787c to 0xb732eff 2024-09-16T13:28:36.442 INFO:teuthology.orchestra.run.smithi076.stdout:308 write 0xa70d796 thru 0xa7191e5 (0xba50 bytes) 2024-09-16T13:28:36.449 INFO:teuthology.orchestra.run.smithi076.stdout:309 read 0x51baf69 thru 0x51bd8b9 (0x2951 bytes) 2024-09-16T13:28:36.449 INFO:teuthology.orchestra.run.smithi076.stdout:310 punch from 0x64e062d to 0x64ee3ca, (0xdd9d bytes) 2024-09-16T13:28:36.449 INFO:teuthology.orchestra.run.smithi076.stdout:311 write 0x26448b3 thru 0x2653cc4 (0xf412 bytes) 2024-09-16T13:28:36.492 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.492 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:36.503 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:36.500+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:36.553 INFO:teuthology.orchestra.run.smithi076.stdout:313 read 0x31ebb2b thru 0x31f7cb3 (0xc189 bytes) 2024-09-16T13:28:36.553 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:36.553 INFO:teuthology.orchestra.run.smithi076.stdout:315 write 0x7ed355a thru 0x7ee2dc0 (0xf867 bytes) 2024-09-16T13:28:36.559 INFO:teuthology.orchestra.run.smithi076.stdout:316 read 0x9005281 thru 0x9009a2b (0x47ab bytes) 2024-09-16T13:28:36.565 INFO:teuthology.orchestra.run.smithi076.stdout:317 write 0xbec4570 thru 0xbecbc3e (0x76cf bytes) 2024-09-16T13:28:36.567 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:36.567 INFO:teuthology.orchestra.run.smithi076.stdout:319 write 0x40412de thru 0x404c2c7 (0xafea bytes) 2024-09-16T13:28:36.572 INFO:teuthology.orchestra.run.smithi076.stdout:320 write 0x6d3e398 thru 0x6d4943a (0xb0a3 bytes) 2024-09-16T13:28:36.574 INFO:teuthology.orchestra.run.smithi076.stdout:321 read 0x30a58b7 thru 0x30aa5ef (0x4d39 bytes) 2024-09-16T13:28:36.575 INFO:teuthology.orchestra.run.smithi076.stdout:322 clone 6 order 20 su 131072 sc 14 2024-09-16T13:28:36.604 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:36.602+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:36.640 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.641 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:36.704 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:36.703+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:36.787 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.788 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.796 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:36.841 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:36.839+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:36.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:36.895 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:36.895 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:36.905 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:36.906 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:36.904+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:36.925 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:28:36.985 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:36.986 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:37.001 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:37.002 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:37.007 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:37.005+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:37.019 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:37.020 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:37.035 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:37.036 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:37.054 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:37.054 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:37.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:37.054 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2221 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.106198+0000", 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9445344759999998, 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093146889999999993, 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2221 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.055 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106198+0000", 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106198+0000", 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106199+0000", 2024-09-16T13:28:37.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0419999999999999e-06 2024-09-16T13:28:37.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106204+0000", 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8779999999999997e-06 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106215+0000", 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.093e-05 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106221+0000", 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7180000000000001e-06 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106269+0000", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8083e-05 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106336+0000", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7148000000000002e-05 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.106640+0000", 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030464700000000001 2024-09-16T13:28:37.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114368+0000", 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077275299999999998 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115431+0000", 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010632770000000001 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115471+0000", 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0191999999999997e-05 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115513+0000", 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1244e-05 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2254 4.14 4:2f26fa94:::rbd_data.11a962dad130.0000000000000032:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.064 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.107490+0000", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9432424990000001, 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090201780000000002, 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2254 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107490+0000", 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107490+0000", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107491+0000", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7700000000000003e-07 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107729+0000", 2024-09-16T13:28:37.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023766600000000001 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107739+0000", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0076e-05 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107744+0000", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107790+0000", 2024-09-16T13:28:37.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6147999999999999e-05 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.107859+0000", 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8639999999999993e-05 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108133+0000", 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027442500000000002 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114405+0000", 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062712139999999998 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116395+0000", 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019904430000000002 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116440+0000", 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5318000000000002e-05 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116510+0000", 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9868000000000003e-05 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2256 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9425608400000001, 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016169003000000001, 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.070 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2256 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8100000000000002e-07 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108176+0000", 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108184+0000", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1239999999999999e-06 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108187+0000", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112601+0000", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044130940000000002 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112684+0000", 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3038e-05 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112976+0000", 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000292665 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116526+0000", 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035497549999999999 2024-09-16T13:28:37.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124246+0000", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00771975 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124298+0000", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2688000000000001e-05 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124341+0000", 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2218999999999997e-05 2024-09-16T13:28:37.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2260 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9413607929999999, 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015260647, 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2024-09-16T13:28:37.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109373+0000", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.11e-06 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109378+0000", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.276e-06 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109392+0000", 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3873000000000001e-05 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109399+0000", 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3900000000000004e-06 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113426+0000", 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040269540000000001 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113497+0000", 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1154000000000003e-05 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114052+0000", 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000554742 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118779+0000", 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047264669999999998 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124446+0000", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056670360000000003 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124495+0000", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8887999999999997e-05 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124632+0000", 2024-09-16T13:28:37.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013775699999999999 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2262 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9400114720000001, 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014235015, 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.082 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110722+0000", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-07 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110727+0000", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8899999999999998e-06 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110737+0000", 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0311e-05 2024-09-16T13:28:37.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110743+0000", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6470000000000003e-06 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114432+0000", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036886760000000001 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114502+0000", 2024-09-16T13:28:37.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0246000000000005e-05 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114821+0000", 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031939499999999999 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118971+0000", 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041499559999999998 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124857+0000", 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005885782 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124912+0000", 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4950000000000001e-05 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124956+0000", 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4165000000000002e-05 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2264 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9385734989999999, 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012912022, 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2264 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112160+0000", 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9199999999999999e-07 2024-09-16T13:28:37.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112164+0000", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3789999999999999e-06 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112175+0000", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.062e-05 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112182+0000", 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2910000000000002e-06 2024-09-16T13:28:37.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115155+0000", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00297255 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115230+0000", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5164000000000001e-05 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125071+0000", 2024-09-16T13:28:37.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098410259999999992 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2314 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.114866+0000", 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9358668219999999, 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010313951, 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.092 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2314 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114866+0000", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114866+0000", 2024-09-16T13:28:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114867+0000", 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2550000000000001e-06 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114871+0000", 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114880+0000", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7800000000000006e-06 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114885+0000", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9849999999999997e-06 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115551+0000", 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00066604200000000004 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115636+0000", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5125999999999998e-05 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115962+0000", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00032548000000000001 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121552+0000", 2024-09-16T13:28:37.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055903630000000001 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125110+0000", 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003558 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125145+0000", 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5434e-05 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125180+0000", 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4272e-05 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2316 4.10 4:0d5a9ded:::rbd_data.11a962dad130.0000000000000035:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116064+0000", 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9346682319999999, 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011791830999999999, 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.098 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2316 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116064+0000", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116064+0000", 2024-09-16T13:28:37.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116065+0000", 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116415+0000", 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000349501 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116424+0000", 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5459999999999997e-06 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116430+0000", 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4489999999999998e-06 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116486+0000", 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5940000000000003e-05 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116561+0000", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5536000000000003e-05 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116770+0000", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020887799999999999 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125848+0000", 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090777730000000008 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.127763+0000", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019150770000000001 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.127818+0000", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4953000000000001e-05 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.127856+0000", 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8024000000000001e-05 2024-09-16T13:28:37.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2317 4.10 4:0d5a9ded:::rbd_data.11a962dad130.0000000000000035:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9342811609999999, 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013390631, 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2317 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116452+0000", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116455+0000", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6510000000000001e-06 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116462+0000", 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9659999999999999e-06 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116466+0000", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121568+0000", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051019619999999998 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121646+0000", 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8288000000000007e-05 2024-09-16T13:28:37.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121822+0000", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017609200000000001 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125896+0000", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040733840000000002 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129420+0000", 2024-09-16T13:28:37.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003524241 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129772+0000", 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035258700000000002 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129842+0000", 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9506999999999997e-05 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2339 4.1d 4:bb7b2676:::rbd_data.11a962dad130.0000000000000039:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.125679+0000", 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9250530750000001, 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010829274999999999, 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2339 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.110 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125679+0000", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125679+0000", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125681+0000", 2024-09-16T13:28:37.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1969999999999999e-06 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125684+0000", 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.129e-06 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125848+0000", 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016414700000000001 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125853+0000", 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129225+0000", 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033720529999999999 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129301+0000", 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5895000000000004e-05 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129517+0000", 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021625099999999999 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.133975+0000", 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044578329999999996 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.136423+0000", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024482000000000002 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.136463+0000", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0028999999999999e-05 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.136509+0000", 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5268000000000001e-05 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.115 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2396 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'5, mlcod=37'3)", 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.920664114, 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013627254, 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130069+0000", 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8999999999999995e-07 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130195+0000", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012528800000000001 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130204+0000", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9259999999999995e-06 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130211+0000", 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.024e-06 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130789+0000", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057756300000000001 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130806+0000", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7314999999999999e-05 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143643+0000", 2024-09-16T13:28:37.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012836172 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143696+0000", 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3075999999999998e-05 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2397 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'7, mlcod=37'3)", 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:37.120 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9183482380000001, 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01459504, 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132385+0000", 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6700000000000003e-07 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132395+0000", 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9620000000000005e-06 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132405+0000", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8600000000000005e-06 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132410+0000", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0579999999999997e-06 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138648+0000", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062382610000000002 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138673+0000", 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779000000000001e-05 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146940+0000", 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082671900000000006 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146979+0000", 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9162999999999998e-05 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2475 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.911053779, 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014446321999999999, 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2475 2024-09-16T13:28:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139680+0000", 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.19e-06 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139685+0000", 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3689999999999998e-06 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139698+0000", 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2376e-05 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139704+0000", 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.105e-06 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140099+0000", 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000395413 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140182+0000", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2818999999999994e-05 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140518+0000", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033604199999999999 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144678+0000", 2024-09-16T13:28:37.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004159991 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154051+0000", 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093730879999999999 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154086+0000", 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4598000000000003e-05 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154125+0000", 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9331e-05 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2499 4.14 4:297d7757:::rbd_data.11a962dad130.0000000000000042:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.144558+0000", 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9061740149999999, 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011690727, 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.131 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2499 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144558+0000", 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144558+0000", 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144559+0000", 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000003e-07 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144612+0000", 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2457000000000003e-05 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144623+0000", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1759999999999999e-05 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144630+0000", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2890000000000003e-06 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144699+0000", 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9008000000000001e-05 2024-09-16T13:28:37.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144782+0000", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3336000000000003e-05 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145145+0000", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036319999999999999 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154171+0000", 2024-09-16T13:28:37.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090260400000000008 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156164+0000", 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019928179999999999 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156211+0000", 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6953999999999998e-05 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156249+0000", 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8111999999999997e-05 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2500 4.14 4:297d7757:::rbd_data.11a962dad130.0000000000000042:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.144665+0000", 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9060677620000002, 2024-09-16T13:28:37.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011718395, 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2500 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144665+0000", 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144665+0000", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144665+0000", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144668+0000", 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8499999999999998e-06 2024-09-16T13:28:37.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144676+0000", 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7980000000000006e-06 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144680+0000", 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2309999999999998e-06 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.149032+0000", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043515649999999999 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.149111+0000", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9647000000000005e-05 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.149324+0000", 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000212363 2024-09-16T13:28:37.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154214+0000", 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048903260000000004 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156309+0000", 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020945529999999999 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156347+0000", 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8685e-05 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.156383+0000", 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5692000000000003e-05 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2525 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9057912020000001, 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012849027000000001, 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2525 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144942+0000", 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1060000000000001e-06 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145045+0000", 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103089 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145058+0000", 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2242000000000001e-05 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145063+0000", 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7760000000000003e-06 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150004+0000", 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049408229999999996 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150079+0000", 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4247999999999998e-05 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150307+0000", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022807200000000001 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157405+0000", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070986110000000003 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157707+0000", 2024-09-16T13:28:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030204199999999998 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157749+0000", 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1573999999999999e-05 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157790+0000", 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1443999999999998e-05 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2526 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.9056618140000001, 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013452830000000001, 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2526 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145074+0000", 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145082+0000", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2009999999999999e-06 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145086+0000", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999996e-06 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154239+0000", 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091526520000000007 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154329+0000", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0061000000000004e-05 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154521+0000", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019246399999999999 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158411+0000", 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038892010000000001 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158452+0000", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1776000000000001e-05 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158487+0000", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4725999999999999e-05 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158523+0000", 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6443000000000003e-05 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2541 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.146726+0000", 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.904006576, 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011972053, 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2541 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146726+0000", 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146726+0000", 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146727+0000", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.083e-06 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146732+0000", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4369999999999998e-06 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146745+0000", 2024-09-16T13:28:37.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2322e-05 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146751+0000", 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.195e-06 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.155461+0000", 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087103219999999995 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.155538+0000", 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736000000000005e-05 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.155858+0000", 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031987600000000003 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158531+0000", 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026730980000000001 2024-09-16T13:28:37.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158622+0000", 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1181000000000007e-05 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158665+0000", 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2524000000000001e-05 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158698+0000", 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3278999999999998e-05 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2676 4.19 4:9d8c3b29:::rbd_data.11a962dad130.000000000000005a:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.166633+0000", 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.8840993790000002, 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097592470000000004, 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2676 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166633+0000", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166633+0000", 2024-09-16T13:28:37.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166634+0000", 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0200000000000001e-07 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166637+0000", 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6009999999999999e-06 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166659+0000", 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1599000000000001e-05 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166665+0000", 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9009999999999999e-06 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166736+0000", 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0646000000000001e-05 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166961+0000", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000225524 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.167139+0000", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017811300000000001 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.171929+0000", 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047895580000000002 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.176314+0000", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043852450000000003 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.176355+0000", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1282999999999998e-05 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.176392+0000", 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6974999999999998e-05 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:3297 4.17 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2069521~10445] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.3834751540000001, 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030440354999999999, 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3297 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667258+0000", 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668628+0000", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001369501 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668651+0000", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3017999999999999e-05 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668658+0000", 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668704+0000", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5479000000000003e-05 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668825+0000", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121791 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.669018+0000", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019308300000000001 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:37.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.685320+0000", 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016301690000000001 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697626+0000", 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012306375 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697668+0000", 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999997e-05 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:37.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697698+0000", 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0006999999999999e-05 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:37.173 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:37.174 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:37.174 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:37.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:37.175 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:37.176 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:37.176 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:37.176 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:37.176 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:37.177 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:37.177 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone5 from 0xbecbc3f (overlap 0x912787c) to 0xe83f3f 2024-09-16T13:28:37.178 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:37.172+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:37.275 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:37.273+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:37.281 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:37.281 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:37.281 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:37.281 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:37.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:37.371 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-09-16T13:28:37.376 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:37.374+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:37.402 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:37.402 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:37.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:37.452 INFO:teuthology.orchestra.run.smithi076.stdout:323 read 0x2fbcf17 thru 0x2fbf412 (0x24fc bytes) 2024-09-16T13:28:37.454 INFO:teuthology.orchestra.run.smithi076.stdout:324 read 0x2625583 thru 0x26270da (0x1b58 bytes) 2024-09-16T13:28:37.462 INFO:teuthology.orchestra.run.smithi076.stdout:329 trunc from 0xbecbc3f to 0x5b758bd 2024-09-16T13:28:37.477 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:37.475+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:37.493 INFO:teuthology.orchestra.run.smithi076.stdout:330 read 0x59c414 thru 0x59e1fd (0x1dea bytes) 2024-09-16T13:28:37.494 INFO:teuthology.orchestra.run.smithi076.stdout:331 punch from 0x554a94e to 0x554e83a, (0x3eec bytes) 2024-09-16T13:28:37.494 INFO:teuthology.orchestra.run.smithi076.stdout:333 punch from 0xb5630d to 0xb5eea2, (0x8b95 bytes) 2024-09-16T13:28:37.495 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.509 INFO:teuthology.orchestra.run.smithi076.stdout:335 write 0xddc56b3 thru 0xddcfbf2 (0xa540 bytes) 2024-09-16T13:28:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:336 read 0x4de62be thru 0x4dea372 (0x40b5 bytes) 2024-09-16T13:28:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:338 read 0x3fba964 thru 0x3fc3ff1 (0x968e bytes) 2024-09-16T13:28:37.527 INFO:teuthology.orchestra.run.smithi076.stdout:339 trunc from 0xddcfbf3 to 0x17acc9a 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3132 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 4368762~22099] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 2.94711831, 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024228047999999999, 2024-09-16T13:28:37.558 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3132 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608311+0000", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4300000000000002e-07 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614081+0000", 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057699559999999997 2024-09-16T13:28:37.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614094+0000", 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2672e-05 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614100+0000", 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.071e-06 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614155+0000", 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614248+0000", 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2958e-05 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614457+0000", 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020964000000000001 2024-09-16T13:28:37.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.629775+0000", 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015317084 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632471+0000", 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026965330000000001 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:37.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632510+0000", 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8831000000000001e-05 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632538+0000", 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8330000000000002e-05 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6310 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1572864~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.564 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.143684+0000", 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.41174401799999999, 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013154242999999999, 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6310 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.565 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143684+0000", 2024-09-16T13:28:37.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143684+0000", 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143685+0000", 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143688+0000", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7020000000000001e-06 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143694+0000", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8009999999999998e-06 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143698+0000", 2024-09-16T13:28:37.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7179999999999998e-06 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.156478+0000", 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012779557 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.156542+0000", 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4702999999999994e-05 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.156839+0000", 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029640800000000001 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6311 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1835008~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.143702+0000", 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.41172655200000002, 2024-09-16T13:28:37.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013557938, 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6311 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143702+0000", 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143702+0000", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143702+0000", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143704+0000", 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:28:37.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143711+0000", 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7390000000000002e-06 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.143714+0000", 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3340000000000002e-06 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.156862+0000", 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013148277 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.156943+0000", 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0742999999999996e-05 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.157260+0000", 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031677499999999999 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6959 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382735+0000", 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17269321500000001, 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013249221, 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6959 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.577 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382735+0000", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382735+0000", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382736+0000", 2024-09-16T13:28:37.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382740+0000", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0389999999999998e-06 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382743+0000", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5999999999999998e-06 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382747+0000", 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:28:37.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.395803+0000", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013056804999999999 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.395880+0000", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6725999999999997e-05 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.395985+0000", 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010429000000000001 2024-09-16T13:28:37.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6961 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382762+0000", 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17266646999999999, 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013266454, 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6961 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382762+0000", 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382762+0000", 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382763+0000", 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3600000000000004e-07 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382764+0000", 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8700000000000001e-06 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382769+0000", 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1150000000000004e-06 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382771+0000", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2740000000000002e-06 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.395840+0000", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013068732 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.395922+0000", 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2095000000000006e-05 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396028+0000", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000106832 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6963 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382773+0000", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17265539399999999, 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013447205, 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.585 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6963 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382773+0000", 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382773+0000", 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382773+0000", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1800000000000002e-07 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382774+0000", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.029e-06 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382778+0000", 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382780+0000", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396048+0000", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013268184000000001 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396123+0000", 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5569999999999996e-05 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396220+0000", 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6812999999999996e-05 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6969 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382781+0000", 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17264703000000001, 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013692635, 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.589 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6969 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382781+0000", 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382781+0000", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382782+0000", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9299999999999999e-07 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382787+0000", 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.853e-06 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382790+0000", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.976e-06 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382792+0000", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3429999999999998e-06 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396258+0000", 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013465816 2024-09-16T13:28:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396348+0000", 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0043999999999994e-05 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396474+0000", 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012631 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.593 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6971 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382795+0000", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.172633394, 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013878115, 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6971 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382795+0000", 2024-09-16T13:28:37.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382795+0000", 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382796+0000", 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1300000000000001e-07 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382797+0000", 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.237e-06 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382800+0000", 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9040000000000001e-06 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382801+0000", 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.792e-06 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396493+0000", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013691333999999999 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396568+0000", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5617000000000005e-05 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396673+0000", 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104818 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.597 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6979 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17259907499999999, 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014097060999999999, 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6979 2024-09-16T13:28:37.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0900000000000001e-07 2024-09-16T13:28:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382835+0000", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6929999999999998e-06 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382842+0000", 2024-09-16T13:28:37.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3640000000000004e-06 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396711+0000", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013869545 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396799+0000", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7874999999999995e-05 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396926+0000", 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012751700000000001 2024-09-16T13:28:37.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6981 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.172568796, 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014344342, 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.602 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6981 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.34e-07 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382861+0000", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.003e-06 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382864+0000", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8430000000000001e-06 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382866+0000", 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8360000000000001e-06 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396969+0000", 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01410347 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397099+0000", 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013018700000000001 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397204+0000", 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104769 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6996 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.172525387, 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014259829, 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.606 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6996 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382904+0000", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5400000000000002e-07 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382905+0000", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382911+0000", 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:28:37.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382914+0000", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3270000000000001e-06 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396956+0000", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014041394 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397050+0000", 2024-09-16T13:28:37.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4258000000000005e-05 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397163+0000", 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113002 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6997 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17251059999999999, 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014462567000000001, 2024-09-16T13:28:37.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6997 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5100000000000001e-07 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382920+0000", 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6950000000000001e-06 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382925+0000", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6619999999999998e-06 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397188+0000", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014258694000000001 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397262+0000", 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4282000000000005e-05 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397380+0000", 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118411 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6998 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17249589100000001, 2024-09-16T13:28:37.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014636431, 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6998 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7199999999999999e-07 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382940+0000", 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0090000000000001e-06 2024-09-16T13:28:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382944+0000", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9670000000000003e-06 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382946+0000", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0549999999999998e-06 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397421+0000", 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014474792 2024-09-16T13:28:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397498+0000", 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6746e-05 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397569+0000", 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1390000000000006e-05 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.618 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7000 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2555904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17247278199999999, 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014812446, 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7000 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.51e-07 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382958+0000", 2024-09-16T13:28:37.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.827e-06 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382964+0000", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9900000000000002e-06 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382967+0000", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397606+0000", 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014638979999999999 2024-09-16T13:28:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397684+0000", 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929000000000006e-05 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397768+0000", 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4515e-05 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7008 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17245971500000001, 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014735491, 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7008 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3599999999999999e-07 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382970+0000", 2024-09-16T13:28:37.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1370000000000001e-06 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382973+0000", 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9979999999999999e-06 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382980+0000", 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0230000000000004e-06 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397590+0000", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014609347 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397630+0000", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0184e-05 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397704+0000", 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4466000000000002e-05 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7011 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.172441342, 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014843745, 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7011 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382988+0000", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382990+0000", 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9810000000000002e-06 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382996+0000", 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3400000000000003e-06 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382999+0000", 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4709999999999999e-06 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397717+0000", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014718072 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397757+0000", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9796999999999999e-05 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397831+0000", 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3656000000000001e-05 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.630 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7023 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17239779399999999, 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015041545999999999, 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7023 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1400000000000001e-07 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383032+0000", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9800000000000002e-07 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383035+0000", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.768e-06 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383038+0000", 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.971e-06 2024-09-16T13:28:37.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397863+0000", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014825155 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397954+0000", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0958999999999994e-05 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398072+0000", 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011848100000000001 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.634 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7024 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17238790400000001, 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015178159, 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7024 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.635 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1899999999999998e-07 2024-09-16T13:28:37.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383042+0000", 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.57e-06 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383047+0000", 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-06 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383050+0000", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398103+0000", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052963000000001 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398151+0000", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8863999999999998e-05 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398219+0000", 2024-09-16T13:28:37.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7291000000000003e-05 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7032 4.1 4:80d51c4d:::rbd_data.11a962dad130.0000000000000016:6 [read 458752~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.384744+0000", 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.17068467900000001, 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013698001, 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7032 2024-09-16T13:28:37.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384744+0000", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384744+0000", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384745+0000", 2024-09-16T13:28:37.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1620000000000001e-06 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384750+0000", 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7350000000000004e-06 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384761+0000", 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0925e-05 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.384767+0000", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1480000000000003e-06 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398257+0000", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013489734 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398347+0000", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0109000000000001e-05 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398442+0000", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5187999999999997e-05 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7038 4.1 4:87ec5c63:::rbd_data.11a962dad130.0000000000000027:6 [read 393216~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.385612+0000", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.169816411, 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013196586, 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:37.643 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7038 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385612+0000", 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385612+0000", 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:37.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385613+0000", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7700000000000005e-07 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385618+0000", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4739999999999996e-06 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385625+0000", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9750000000000001e-06 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:37.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.385629+0000", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1200000000000004e-06 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398623+0000", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012994036 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398699+0000", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5663999999999999e-05 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398809+0000", 2024-09-16T13:28:37.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010984000000000001 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:37.647 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:37.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:37.649 INFO:teuthology.orchestra.run.smithi076.stdout:340 punch from 0x92a723 to 0x938474, (0xdd51 bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:341 trunc from 0x17acc9a to 0xd38bbed 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:342 write 0x45332ee thru 0x4538cd7 (0x59ea bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:344 punch from 0x54763e8 to 0x547b5d8, (0x51f0 bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:345 write 0x7b9d032 thru 0x7ba8f7b (0xbf4a bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:346 write 0x7c7f3ee thru 0x7c84aa2 (0x56b5 bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:348 write 0xd1d0306 thru 0xd1d284b (0x2546 bytes) 2024-09-16T13:28:37.650 INFO:teuthology.orchestra.run.smithi076.stdout:350 trunc from 0xd38bbed to 0xbda8672 2024-09-16T13:28:37.650 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:37.646+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:37.654 INFO:teuthology.orchestra.run.smithi076.stdout:351 writesame 0xe1df57f thru 0xe1e63c4 data_size 0x3(0x6e46 bytes) 2024-09-16T13:28:37.654 INFO:teuthology.orchestra.run.smithi076.stdout:352 punch from 0xdd91f24 to 0xdd9d653, (0xb72f bytes) 2024-09-16T13:28:37.655 INFO:teuthology.orchestra.run.smithi076.stdout:355 punch from 0x1fccc72 to 0x1fd0146, (0x34d4 bytes) 2024-09-16T13:28:37.655 INFO:teuthology.orchestra.run.smithi076.stdout:357 write 0xd318f5e thru 0xd31a569 (0x160c bytes) 2024-09-16T13:28:37.657 INFO:teuthology.orchestra.run.smithi076.stdout:358 read 0x313d25f thru 0x31426d8 (0x547a bytes) 2024-09-16T13:28:37.657 INFO:teuthology.orchestra.run.smithi076.stdout:359 writesame 0x166b112 thru 0x16723c9 data_size 0x4(0x72b8 bytes) 2024-09-16T13:28:37.674 INFO:teuthology.orchestra.run.smithi076.stdout:361 trunc from 0xe1e63c5 to 0x9f41845 2024-09-16T13:28:37.684 INFO:teuthology.orchestra.run.smithi076.stdout:362 punch from 0x6d4616c to 0x6d49385, (0x3219 bytes) 2024-09-16T13:28:37.684 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.684 INFO:teuthology.orchestra.run.smithi076.stdout:365 read 0x54ce60b thru 0x54db99b (0xd391 bytes) 2024-09-16T13:28:37.684 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xdb95ef8 2024-09-16T13:28:37.689 INFO:teuthology.orchestra.run.smithi076.stdout:366 trunc from 0x9f41845 to 0xdb95ef8 2024-09-16T13:28:37.694 INFO:teuthology.orchestra.run.smithi076.stdout:367 punch from 0x499486c to 0x499b1e0, (0x6974 bytes) 2024-09-16T13:28:37.694 INFO:teuthology.orchestra.run.smithi076.stdout:369 read 0x44d2b42 thru 0x44da8a4 (0x7d63 bytes) 2024-09-16T13:28:37.694 INFO:teuthology.orchestra.run.smithi076.stdout:370 write 0x82ab499 thru 0x82b64ff (0xb067 bytes) 2024-09-16T13:28:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:371 read 0x721f75a thru 0x7221341 (0x1be8 bytes) 2024-09-16T13:28:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:372 read 0x4c6c0ac thru 0x4c713d8 (0x532d bytes) 2024-09-16T13:28:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:374 read 0x3500744 thru 0x350d814 (0xd0d1 bytes) 2024-09-16T13:28:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:375 write 0x735f7f6 thru 0x73684cb (0x8cd6 bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:376 punch from 0x88e9f87 to 0x88f5f0f, (0xbf88 bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:377 read 0xa602df2 thru 0xa60f417 (0xc626 bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:379 read 0x279e409 thru 0x27acbf8 (0xe7f0 bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:381 read 0x5276daf thru 0x52777a3 (0x9f5 bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:382 punch from 0xc4088c7 to 0xc40f7c3, (0x6efc bytes) 2024-09-16T13:28:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:383 write 0xa9078d0 thru 0xa9152df (0xda10 bytes) 2024-09-16T13:28:37.709 INFO:teuthology.orchestra.run.smithi076.stdout:384 punch from 0x7c07e14 to 0x7c17c73, (0xfe5f bytes) 2024-09-16T13:28:37.709 INFO:teuthology.orchestra.run.smithi076.stdout:385 writesame 0x92b6b91 thru 0x92c022f data_size 0x1(0x969f bytes) 2024-09-16T13:28:37.748 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:37.747+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:37.753 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:37.753 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:37.753 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:37.753 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:37.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:37.783 INFO:teuthology.orchestra.run.smithi076.stdout:388 read 0x4f2baa0 thru 0x4f2ff15 (0x4476 bytes) 2024-09-16T13:28:37.783 INFO:teuthology.orchestra.run.smithi076.stdout:389 write 0xcfa154f thru 0xcfaf805 (0xe2b7 bytes) 2024-09-16T13:28:37.797 INFO:teuthology.orchestra.run.smithi076.stdout:390 trunc from 0xdb95ef8 to 0x59df9c9 2024-09-16T13:28:37.823 INFO:teuthology.orchestra.run.smithi076.stdout:391 trunc from 0x59df9c9 to 0xd71df80 2024-09-16T13:28:37.836 INFO:teuthology.orchestra.run.smithi076.stdout:392 trunc from 0xd71df80 to 0x814c71f 2024-09-16T13:28:37.841 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.841 INFO:teuthology.orchestra.run.smithi076.stdout:395 write 0x2319c23 thru 0x2324da1 (0xb17f bytes) 2024-09-16T13:28:37.848 INFO:teuthology.orchestra.run.smithi076.stdout:396 writesame 0x58a012a thru 0x58a4ef0 data_size 0x1(0x4dc7 bytes) 2024-09-16T13:28:37.849 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:37.847+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:37.866 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:37.867 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:37.867 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:37.867 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:37.867 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:37.869 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:37.869 INFO:teuthology.orchestra.run.smithi076.stdout:400 read 0x79638de thru 0x7965fe9 (0x270c bytes) 2024-09-16T13:28:37.869 INFO:teuthology.orchestra.run.smithi076.stdout:401 punch from 0xea3e9b to 0xeaf553, (0xb6b8 bytes) 2024-09-16T13:28:37.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:37.881 INFO:teuthology.orchestra.run.smithi076.stdout:403 write 0xdb8a1a4 thru 0xdb8df31 (0x3d8e bytes) 2024-09-16T13:28:37.885 INFO:teuthology.orchestra.run.smithi076.stdout:404 read 0x7daa680 thru 0x7db719d (0xcb1e bytes) 2024-09-16T13:28:37.885 INFO:teuthology.orchestra.run.smithi076.stdout:405 write 0x8d3c7c5 thru 0x8d49769 (0xcfa5 bytes) 2024-09-16T13:28:37.891 INFO:teuthology.orchestra.run.smithi076.stdout:406 writesame 0xa540b1e thru 0xa54ccf5 data_size 0x4(0xc1d8 bytes) 2024-09-16T13:28:37.909 INFO:teuthology.orchestra.run.smithi076.stdout:407 writesame 0xe3c8719 thru 0xe3cbca7 data_size 0x1(0x358f bytes) 2024-09-16T13:28:37.909 INFO:teuthology.orchestra.run.smithi076.stdout:408 write 0x1212426 thru 0x1217f8b (0x5b66 bytes) 2024-09-16T13:28:37.911 INFO:teuthology.orchestra.run.smithi076.stdout:410 read 0x97496f3 thru 0x974e111 (0x4a1f bytes) 2024-09-16T13:28:37.911 INFO:teuthology.orchestra.run.smithi076.stdout:411 writesame 0x263e3dd thru 0x2646658 data_size 0xe(0x827c bytes) 2024-09-16T13:28:37.915 INFO:teuthology.orchestra.run.smithi076.stdout:412 write 0x834f4b7 thru 0x834f5d5 (0x11f bytes) 2024-09-16T13:28:37.918 INFO:teuthology.orchestra.run.smithi076.stdout:413 writesame 0xdc99568 thru 0xdca3da7 data_size 0x8(0xa840 bytes) 2024-09-16T13:28:37.928 INFO:teuthology.orchestra.run.smithi076.stdout:416 writesame 0xc59f4ff thru 0xc5acc69 data_size 0x1(0xd76b bytes) 2024-09-16T13:28:37.950 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:37.948+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:38.013 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:38.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:28:38.013 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T13:28:38.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:28:38.017 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.017 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.374916566, 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:28:38.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.284386161, 2024-09-16T13:28:38.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:28:38.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3536 4.3 e38/32 4:c24635eb:::rbd_data.11a976a8123d.0000000000000000:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2752254629999999, 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018037721, 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739179+0000", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.316e-06 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742916+0000", 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037375210000000002 2024-09-16T13:28:38.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742931+0000", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5413999999999998e-05 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742941+0000", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4290000000000007e-06 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742984+0000", 2024-09-16T13:28:38.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3022000000000002e-05 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742999+0000", 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5279000000000001e-05 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757171+0000", 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014172034999999999 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757215+0000", 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3705000000000003e-05 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2572569910000002, 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:28:38.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.84149483899999999, 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:28:38.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.77115855700000002, 2024-09-16T13:28:38.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:28:38.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6939 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 983040~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63179487000000001, 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017717554999999999, 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.047 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6939 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382610+0000", 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4330000000000001e-06 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382616+0000", 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382620+0000", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6459999999999999e-06 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400226+0000", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017606218 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400267+0000", 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0914000000000002e-05 2024-09-16T13:28:38.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400325+0000", 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8378999999999998e-05 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6947 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63177193099999995, 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017826794, 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6947 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.03e-07 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382633+0000", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0990000000000001e-06 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382641+0000", 2024-09-16T13:28:38.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3889999999999999e-06 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382644+0000", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7740000000000002e-06 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400347+0000", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017702215 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400390+0000", 2024-09-16T13:28:38.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3071999999999997e-05 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400458+0000", 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7842000000000005e-05 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6950 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63175424999999996, 2024-09-16T13:28:38.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017927169999999999, 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6950 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382649+0000", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.41e-07 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382656+0000", 2024-09-16T13:28:38.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8220000000000001e-06 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382663+0000", 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1210000000000001e-06 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382666+0000", 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.534e-06 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400478+0000", 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017811421000000001 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400519+0000", 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0964999999999998e-05 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400576+0000", 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6966000000000003e-05 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6960 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.631732513, 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018031049, 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6960 2024-09-16T13:28:38.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382671+0000", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382672+0000", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.601e-06 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382678+0000", 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382681+0000", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400597+0000", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017916527000000002 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400640+0000", 2024-09-16T13:28:38.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2394000000000001e-05 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400701+0000", 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1334000000000006e-05 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6962 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:38.064 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63171783999999997, 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018139466, 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6962 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382687+0000", 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382694+0000", 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7800000000000003e-06 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382697+0000", 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400722+0000", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018024563 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400763+0000", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1003e-05 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400824+0000", 2024-09-16T13:28:38.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1496999999999998e-05 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6968 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63170190699999995, 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018252758000000001, 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.069 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6968 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382703+0000", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382712+0000", 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9600000000000006e-06 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382715+0000", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5640000000000001e-06 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400844+0000", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018129125999999999 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400893+0000", 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8869000000000003e-05 2024-09-16T13:28:38.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400954+0000", 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.02e-05 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6970 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63168258399999999, 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018372506, 2024-09-16T13:28:38.073 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6970 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382722+0000", 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382728+0000", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382751+0000", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3697e-05 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400983+0000", 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018231872999999999 2024-09-16T13:28:38.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401028+0000", 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5148999999999998e-05 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401093+0000", 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4222999999999998e-05 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.078 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6984 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.078 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:38.078 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63154857200000003, 2024-09-16T13:28:38.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018365686999999999, 2024-09-16T13:28:38.080 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.080 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.080 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.080 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6984 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382855+0000", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.06e-07 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382858+0000", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382868+0000", 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9450000000000005e-06 2024-09-16T13:28:38.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382877+0000", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3829999999999998e-06 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401119+0000", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018241384999999999 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401160+0000", 2024-09-16T13:28:38.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1393999999999997e-05 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401220+0000", 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9558999999999997e-05 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6986 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:38.084 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63151974499999997, 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018458407, 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6986 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382886+0000", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.159e-06 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382892+0000", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3690000000000003e-06 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382896+0000", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401242+0000", 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018346475000000001 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401282+0000", 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401341+0000", 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8906999999999997e-05 2024-09-16T13:28:38.088 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6990 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63150269599999997, 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018563608999999998, 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.089 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6990 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382902+0000", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382909+0000", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382913+0000", 2024-09-16T13:28:38.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.552e-06 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401365+0000", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018452196000000001 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401405+0000", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0525999999999998e-05 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401464+0000", 2024-09-16T13:28:38.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8226000000000001e-05 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6991 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2555904~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63148584500000005, 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018669403000000001, 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.093 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6991 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382919+0000", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.671e-06 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382926+0000", 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7460000000000003e-06 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401486+0000", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018556492000000001 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401525+0000", 2024-09-16T13:28:38.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9338999999999997e-05 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401586+0000", 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1124000000000006e-05 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7002 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63146956600000004, 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018778276, 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7002 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382942+0000", 2024-09-16T13:28:38.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4149999999999999e-06 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382949+0000", 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2840000000000001e-06 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382953+0000", 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401609+0000", 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018656005999999999 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401650+0000", 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1077e-05 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401711+0000", 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1245000000000001e-05 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.101 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7003 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63144559899999997, 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018881326, 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.102 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7003 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382959+0000", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.61e-06 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382977+0000", 2024-09-16T13:28:38.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7997999999999999e-05 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382985+0000", 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929999999999998e-06 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401733+0000", 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018748602 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401775+0000", 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1751000000000001e-05 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401838+0000", 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3188000000000006e-05 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7028 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.106 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.63050590799999995, 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018078818999999999, 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7028 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383898+0000", 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0249999999999999e-06 2024-09-16T13:28:38.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383902+0000", 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.357e-06 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383912+0000", 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0356999999999999e-05 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383918+0000", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3290000000000001e-06 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401861+0000", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017943489999999999 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401916+0000", 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4722999999999998e-05 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401976+0000", 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537999999999998e-05 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.111 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.112 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:28:38.113 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.113 INFO:teuthology.orchestra.run.smithi076.stdout:419 read 0x57d0293 thru 0x57da922 (0xa690 bytes) 2024-09-16T13:28:38.113 INFO:teuthology.orchestra.run.smithi076.stdout:420 read 0xdb05dfc thru 0xdb0fb43 (0x9d48 bytes) 2024-09-16T13:28:38.113 INFO:teuthology.orchestra.run.smithi076.stdout:421 trunc from 0xe3cbca8 to 0x8d66e93 2024-09-16T13:28:38.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:423 write 0xc56968e thru 0xc56d83e (0x41b1 bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:424 write 0xc6f0364 thru 0xc6f7d20 (0x79bd bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:425 write 0xb36388d thru 0xb36fefe (0xc672 bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:426 trunc from 0xc6f7d21 to 0x7dd8ee8 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:427 writesame 0x1e48aca thru 0x1e52d4a data_size 0x31(0xa281 bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:429 write 0x8910fe6 thru 0x891e353 (0xd36e bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:430 read 0x55b5292 thru 0x55c26a0 (0xd40f bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:432 read 0x33994a4 thru 0x33a30be (0x9c1b bytes) 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.116 INFO:teuthology.orchestra.run.smithi076.stdout:436 read 0x254c034 thru 0x2553173 (0x7140 bytes) 2024-09-16T13:28:38.117 INFO:teuthology.orchestra.run.smithi076.stdout:437 trunc from 0x891e354 to 0x97a9949 2024-09-16T13:28:38.117 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.117 INFO:teuthology.orchestra.run.smithi076.stdout:439 trunc from 0x97a9949 to 0xbe610da 2024-09-16T13:28:38.117 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:38.110+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:38.117 INFO:teuthology.orchestra.run.smithi076.stdout:440 read 0x38d818c thru 0x38ddd47 (0x5bbc bytes) 2024-09-16T13:28:38.117 INFO:teuthology.orchestra.run.smithi076.stdout:441 read 0x5683c15 thru 0x5686a19 (0x2e05 bytes) 2024-09-16T13:28:38.120 INFO:teuthology.orchestra.run.smithi076.stdout:443 write 0xe77f0d6 thru 0xe7810c6 (0x1ff1 bytes) 2024-09-16T13:28:38.124 INFO:teuthology.orchestra.run.smithi076.stdout:444 writesame 0xa542b50 thru 0xa54b7e1 data_size 0x13(0x8c92 bytes) 2024-09-16T13:28:38.128 INFO:teuthology.orchestra.run.smithi076.stdout:445 write 0x8ce580 thru 0x8d5d09 (0x778a bytes) 2024-09-16T13:28:38.136 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.136 INFO:teuthology.orchestra.run.smithi076.stdout:449 read 0xc5dfb4 thru 0xc675ae (0x95fb bytes) 2024-09-16T13:28:38.144 INFO:teuthology.orchestra.run.smithi076.stdout:450 trunc from 0xe7810c7 to 0x8c47b74 2024-09-16T13:28:38.153 INFO:teuthology.orchestra.run.smithi076.stdout:451 writesame 0x10ca7cd thru 0x10cc93f data_size 0x1(0x2173 bytes) 2024-09-16T13:28:38.171 INFO:teuthology.orchestra.run.smithi076.stdout:452 writesame 0xe0ea3cc thru 0xe0eaf11 data_size 0x1a(0xb46 bytes) 2024-09-16T13:28:38.171 INFO:teuthology.orchestra.run.smithi076.stdout:453 punch from 0xdaad03 to 0xdb848c, (0xd789 bytes) 2024-09-16T13:28:38.171 INFO:teuthology.orchestra.run.smithi076.stdout:454 read 0xc41fa2 thru 0xc4c8ce (0xa92d bytes) 2024-09-16T13:28:38.171 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xdda5507 2024-09-16T13:28:38.172 INFO:teuthology.orchestra.run.smithi076.stdout:455 trunc from 0xe0eaf12 to 0xdda5507 2024-09-16T13:28:38.176 INFO:teuthology.orchestra.run.smithi076.stdout:456 punch from 0x3ec1afe to 0x3ec58f7, (0x3df9 bytes) 2024-09-16T13:28:38.176 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.176 INFO:teuthology.orchestra.run.smithi076.stdout:458 write 0x4d53501 thru 0x4d61c56 (0xe756 bytes) 2024-09-16T13:28:38.194 INFO:teuthology.orchestra.run.smithi076.stdout:460 trunc from 0xdda5507 to 0x374d08c 2024-09-16T13:28:38.207 INFO:teuthology.orchestra.run.smithi076.stdout:462 read 0x18c7e05 thru 0x18cf9ed (0x7be9 bytes) 2024-09-16T13:28:38.207 INFO:teuthology.orchestra.run.smithi076.stdout:463 trunc from 0x374d08c to 0x3cb8af3 2024-09-16T13:28:38.213 INFO:teuthology.orchestra.run.smithi076.stdout:465 punch from 0x183f21e to 0x184d1b9, (0xdf9b bytes) 2024-09-16T13:28:38.213 INFO:teuthology.orchestra.run.smithi076.stdout:469 punch from 0x330b429 to 0x3316cf4, (0xb8cb bytes) 2024-09-16T13:28:38.213 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:38.211+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:38.219 INFO:teuthology.orchestra.run.smithi076.stdout:470 write 0x521d819 thru 0x52204e3 (0x2ccb bytes) 2024-09-16T13:28:38.222 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.222 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:38.222 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:38.222 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:38.232 INFO:teuthology.orchestra.run.smithi076.stdout:471 trunc from 0x52204e4 to 0x9f6db8c 2024-09-16T13:28:38.237 INFO:teuthology.orchestra.run.smithi076.stdout:472 writesame 0x3ac39d9 thru 0x3accfb3 data_size 0xd(0x95db bytes) 2024-09-16T13:28:38.241 INFO:teuthology.orchestra.run.smithi076.stdout:474 punch from 0x36da062 to 0x36e823d, (0xe1db bytes) 2024-09-16T13:28:38.242 INFO:teuthology.orchestra.run.smithi076.stdout:475 read 0x480a032 thru 0x48110dc (0x70ab bytes) 2024-09-16T13:28:38.242 INFO:teuthology.orchestra.run.smithi076.stdout:476 write 0x6525193 thru 0x6533013 (0xde81 bytes) 2024-09-16T13:28:38.250 INFO:teuthology.orchestra.run.smithi076.stdout:477 trunc from 0x9f6db8c to 0x775816a 2024-09-16T13:28:38.263 INFO:teuthology.orchestra.run.smithi076.stdout:478 writesame 0x980eeb2 thru 0x981e8d7 data_size 0x27(0xfa26 bytes) 2024-09-16T13:28:38.274 INFO:teuthology.orchestra.run.smithi076.stdout:480 trunc from 0x981e8d8 to 0x2f8918 2024-09-16T13:28:38.314 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:38.312+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 42, 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:28:38.009692+0000", 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:27:52.988177+0000", 2024-09-16T13:28:38.328 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:28:38.329 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 4, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:38.330 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:38.331 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:38.332 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:28:38.333 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:38.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:38.337 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:28:38.338 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:38.339 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:38.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.341 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:38.342 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.343 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:28:38.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:38.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:38.346 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:38.347 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "pool_client.0", 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:25.583589+0000", 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:28:38.348 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:28:38.349 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "41", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:28:38.350 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 9, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 41, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:28:38.351 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:28:38.352 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.353 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.354 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.357 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.360 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.363 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.364 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:28:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.6766510009765625, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.366 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.369 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:28:38.371 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.372 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:28:38.377 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.378 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.381 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:28:38.383 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.384 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.385 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:28:38.389 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:28:38.390 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 33, 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.393 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.394 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.395 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 19, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:28:38.396 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:28:38.398 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:28:38.401 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.404 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.405 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.406 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.407 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.5", 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [], 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:28:38.408 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [], 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:28:38.409 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:28:38.410 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:28:38.410 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:28:38.410 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:28:38.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:38.410 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:38.411 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.411 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:38.412 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:38.412 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:38.412 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.413 INFO:tasks.thrashosds.thrasher:j is {'epoch': 42, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:28:38.009692+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:27:52.988177+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:28:25.583589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 9, 'snap_epoch': 41, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6766510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 33, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 19, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [6, 1]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:28:38.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:28:38.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:38.414 INFO:teuthology.orchestra.run.smithi076.stdout:481 write 0xaedebc6 thru 0xaee2ab1 (0x3eec bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:482 read 0x5ba28fb thru 0x5ba77cf (0x4ed5 bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:484 write 0x2c498a5 thru 0x2c4bfa7 (0x2703 bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:485 write 0xd64dbbe thru 0xd64ed15 (0x1158 bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:486 write 0x2623112 thru 0x2632eb1 (0xfda0 bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:488 writesame 0xc4173cc thru 0xc41de77 data_size 0x2(0x6aac bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:489 read 0x5975a2d thru 0x597af2e (0x5502 bytes) 2024-09-16T13:28:38.415 INFO:teuthology.orchestra.run.smithi076.stdout:490 write 0xaa69c83 thru 0xaa6c88d (0x2c0b bytes) 2024-09-16T13:28:38.417 INFO:teuthology.orchestra.run.smithi076.stdout:493 write 0x406cd69 thru 0x4079a5d (0xccf5 bytes) 2024-09-16T13:28:38.421 INFO:teuthology.orchestra.run.smithi076.stdout:495 read 0x5559de0 thru 0x556482d (0xaa4e bytes) 2024-09-16T13:28:38.421 INFO:teuthology.orchestra.run.smithi076.stdout:496 write 0x1366daf thru 0x1367c74 (0xec6 bytes) 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:498 read 0xc327905 thru 0xc335fd4 (0xe6d0 bytes) 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:499 read 0x7a7f901 thru 0x7a84f52 (0x5652 bytes) 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.424 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.430 INFO:teuthology.orchestra.run.smithi076.stdout:504 trunc from 0xd64ed16 to 0x83f8f1e 2024-09-16T13:28:38.439 INFO:teuthology.orchestra.run.smithi076.stdout:505 read 0x68777e5 thru 0x687ff3c (0x8758 bytes) 2024-09-16T13:28:38.439 INFO:teuthology.orchestra.run.smithi076.stdout:507 read 0x59db5f2 thru 0x59deec7 (0x38d6 bytes) 2024-09-16T13:28:38.445 INFO:teuthology.orchestra.run.smithi076.stdout:508 trunc from 0x83f8f1e to 0x30e88f3 2024-09-16T13:28:38.453 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:38.451+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:38.455 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.455 INFO:teuthology.orchestra.run.smithi076.stdout:510 punch from 0x6ff932 to 0x700123, (0x7f1 bytes) 2024-09-16T13:28:38.455 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.455 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.456 INFO:teuthology.orchestra.run.smithi076.stdout:513 write 0xf47710 thru 0xf49bb0 (0x24a1 bytes) 2024-09-16T13:28:38.476 INFO:teuthology.orchestra.run.smithi076.stdout:514 write 0xd6ecbb8 thru 0xd6ece0f (0x258 bytes) 2024-09-16T13:28:38.485 INFO:teuthology.orchestra.run.smithi076.stdout:515 trunc from 0xd6ece10 to 0x84f2661 2024-09-16T13:28:38.496 INFO:teuthology.orchestra.run.smithi076.stdout:516 punch from 0x5e30f01 to 0x5e36768, (0x5867 bytes) 2024-09-16T13:28:38.496 INFO:teuthology.orchestra.run.smithi076.stdout:517 write 0x8169ef3 thru 0x817098f (0x6a9d bytes) 2024-09-16T13:28:38.499 INFO:teuthology.orchestra.run.smithi076.stdout:518 punch from 0xd9ce38 to 0xda7633, (0xa7fb bytes) 2024-09-16T13:28:38.500 INFO:teuthology.orchestra.run.smithi076.stdout:519 write 0x52aeffd thru 0x52b6fbb (0x7fbf bytes) 2024-09-16T13:28:38.506 INFO:teuthology.orchestra.run.smithi076.stdout:520 trunc from 0x84f2661 to 0xa5f0702 2024-09-16T13:28:38.517 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:38.515+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:38.523 INFO:teuthology.orchestra.run.smithi076.stdout:522 trunc from 0xa5f0702 to 0xd1a08b 2024-09-16T13:28:38.540 INFO:teuthology.orchestra.run.smithi076.stdout:523 punch from 0x291982 to 0x29a164, (0x87e2 bytes) 2024-09-16T13:28:38.553 INFO:teuthology.orchestra.run.smithi076.stdout:524 trunc from 0xd1a08b to 0xc163a3e 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:38.557 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.8672874329999996, 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:28:38.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:28:38.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.8577159549999998, 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:28:38.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:28:38.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7493 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10027008~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.0103428720000001, 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036308945000000002, 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.567 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7493 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0200000000000001e-07 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543971+0000", 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2599999999999999e-06 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543975+0000", 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9509999999999999e-06 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543977+0000", 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.902e-06 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.579991+0000", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036013993000000001 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580036+0000", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5521e-05 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580273+0000", 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000236116 2024-09-16T13:28:38.571 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7494 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10944512~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.010327491, 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036610563999999998, 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.572 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7494 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543980+0000", 2024-09-16T13:28:38.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.85e-07 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543983+0000", 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543985+0000", 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580303+0000", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036318691 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580359+0000", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5072999999999999e-05 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580590+0000", 2024-09-16T13:28:38.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023103699999999999 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.010319543, 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.577 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.010307273, 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.582 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:28:38.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.010296428, 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.587 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:28:38.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:28:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.010287014, 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.592 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:28:38.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:28:38.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.596 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8153 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26406390099999999, 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036085006000000003, 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8153 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.597 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.99e-07 2024-09-16T13:28:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290244+0000", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290247+0000", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7859999999999998e-06 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290249+0000", 2024-09-16T13:28:38.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.809e-06 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.325803+0000", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035554047999999998 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.325925+0000", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012166499999999999 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326328+0000", 2024-09-16T13:28:38.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040269200000000001 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8154 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26405555400000003, 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036367568000000003, 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8154 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290252+0000", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4799999999999997e-07 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290255+0000", 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9239999999999999e-06 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290257+0000", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7859999999999999e-06 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326341+0000", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036084789999999999 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326384+0000", 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509999999999998e-05 2024-09-16T13:28:38.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326619+0000", 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023450899999999999 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26404788099999998, 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:28:38.605 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.609 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26403953600000002, 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:38.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:28:38.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26402884199999999, 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:28:38.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:38.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.264019854, 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.618 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:38.622 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26401211299999999, 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:38.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:38.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26400355399999997, 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.631 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26399467700000001, 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.26398081200000001, 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.263972345, 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.640 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:28:38.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:28:38.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:38.644 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.263963542, 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:38.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:28:38.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:28:38.648 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.649 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:38.649 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:38.649 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:528 read 0x798fe77 thru 0x79930f6 (0x3280 bytes) 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:529 trunc from 0xc163a3e to 0xbbdb416 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:533 trunc from 0xbbdb416 to 0x8ec1edb 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:534 trunc from 0x8ec1edb to 0x5730238 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:535 read 0x30e435a thru 0x30f2efb (0xeba2 bytes) 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:536 trunc from 0x5730238 to 0xa19eecf 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:537 punch from 0x5abe243 to 0x5ac895b, (0xa718 bytes) 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:538 punch from 0x7cacf2d to 0x7cb43ee, (0x74c1 bytes) 2024-09-16T13:28:38.653 INFO:teuthology.orchestra.run.smithi076.stdout:539 read 0x8b52d6f thru 0x8b55874 (0x2b06 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:540 write 0xdd1b6f8 thru 0xdd2a15c (0xea65 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:542 read 0x32ab2d8 thru 0x32b3b8b (0x88b4 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:543 write 0x7a2128a thru 0x7a2cccf (0xba46 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:544 punch from 0x5abf1eb to 0x5acc5d9, (0xd3ee bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:545 read 0x6d466e7 thru 0x6d4aba0 (0x44ba bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:546 punch from 0x4a42937 to 0x4a4d4f8, (0xabc1 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:547 writesame 0xa8589a5 thru 0xa85a873 data_size 0xb(0x1ecf bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:549 trunc from 0xdd2a15d to 0xd2cf0f8 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:551 writesame 0xe80d9ed thru 0xe81093f data_size 0x1(0x2f53 bytes) 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.654 INFO:teuthology.orchestra.run.smithi076.stdout:553 write 0x7c80f75 thru 0x7c8973e (0x87ca bytes) 2024-09-16T13:28:38.655 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:38.648+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:38.655 INFO:teuthology.orchestra.run.smithi076.stdout:554 trunc from 0xe810940 to 0xd5a5c9d 2024-09-16T13:28:38.658 INFO:teuthology.orchestra.run.smithi076.stdout:555 writesame 0xbfa4f19 thru 0xbfb11ee data_size 0x33(0xc2d6 bytes) 2024-09-16T13:28:38.670 INFO:teuthology.orchestra.run.smithi076.stdout:556 trunc from 0xd5a5c9d to 0x60d0758 2024-09-16T13:28:38.694 INFO:teuthology.orchestra.run.smithi076.stdout:557 writesame 0xbb4e1bf thru 0xbb5420b data_size 0x1(0x604d bytes) 2024-09-16T13:28:38.695 INFO:teuthology.orchestra.run.smithi076.stdout:559 write 0x9d43ebe thru 0x9d52b4d (0xec90 bytes) 2024-09-16T13:28:38.701 INFO:teuthology.orchestra.run.smithi076.stdout:560 read 0x3f99e9 thru 0x40584a (0xbe62 bytes) 2024-09-16T13:28:38.701 INFO:teuthology.orchestra.run.smithi076.stdout:561 writesame 0x97ad6f8 thru 0x97bd1f0 data_size 0x1(0xfaf9 bytes) 2024-09-16T13:28:38.714 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:28:38.715 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:28:38.726 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-16T13:28:38.013439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98529223,"num_objects":132,"num_object_clones":31,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":33,"num_read":3706,"num_read_kb":11886,"num_write":868,"num_write_kb":11337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":693,"ondisk_log_size":693,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2396536,"kb_used_data":5212,"kb_used_omap":13,"kb_used_meta":2391218,"kb_avail":752578184,"statfs":{"total":773094113280,"available":770640060416,"internally_reserved":0,"allocated":5337088,"data_stored":5597853,"data_compressed":991201,"data_compressed_allocated":1236992,"data_compressed_original":3801088,"omap_allocated":13632,"internal_metadata":2448607936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":65986424,"num_objects":101,"num_object_clones":26,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":31,"num_read":3150,"num_read_kb":11350,"num_write":667,"num_write_kb":8825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015262"},"pg_stats":[{"pgid":"4.1f","version":"36'7","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.000356+0000","last_change":"2024-09-16T13:28:28.989963+0000","last_active":"2024-09-16T13:28:33.000356+0000","last_peered":"2024-09-16T13:28:33.000356+0000","last_clean":"2024-09-16T13:28:33.000356+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:28:33.000356+0000","last_undegraded":"2024-09-16T13:28:33.000356+0000","last_fullsized":"2024-09-16T13:28:33.000356+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089281099999999997,"stat_sum":{"num_bytes":3711453,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":5,"num_write":7,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1e","version":"37'1","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.022250+0000","last_change":"2024-09-16T13:28:30.062787+0000","last_active":"2024-09-16T13:28:33.022250+0000","last_peered":"2024-09-16T13:28:33.022250+0000","last_clean":"2024-09-16T13:28:33.022250+0000","last_became_active":"2024-09-16T13:28:26.988624+0000","last_became_peered":"2024-09-16T13:28:26.988624+0000","last_unstale":"2024-09-16T13:28:33.022250+0000","last_undegraded":"2024-09-16T13:28:33.022250+0000","last_fullsized":"2024-09-16T13:28:33.022250+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:44.049188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041021900000000002,"stat_sum":{"num_bytes":49,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1d","version":"38'11","reported_seq":30,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.132170+0000","last_change":"2024-09-16T13:28:28.984841+0000","last_active":"2024-09-16T13:28:34.132170+0000","last_peered":"2024-09-16T13:28:34.132170+0000","last_clean":"2024-09-16T13:28:34.132170+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:28:34.132170+0000","last_undegraded":"2024-09-16T13:28:34.132170+0000","last_fullsized":"2024-09-16T13:28:34.132170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071631099999999997,"stat_sum":{"num_bytes":2386144,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":7,"num_read_kb":128,"num_write":12,"num_write_kb":977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1c","version":"37'6","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.031040+0000","last_change":"2024-09-16T13:28:28.981596+0000","last_active":"2024-09-16T13:28:33.031040+0000","last_peered":"2024-09-16T13:28:33.031040+0000","last_clean":"2024-09-16T13:28:33.031040+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:28:33.031040+0000","last_undegraded":"2024-09-16T13:28:33.031040+0000","last_fullsized":"2024-09-16T13:28:33.031040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030921799999999999,"stat_sum":{"num_bytes":3000958,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":13,"num_write":9,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1b","version":"37'5","reported_seq":23,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.564583+0000","last_change":"2024-09-16T13:28:28.979438+0000","last_active":"2024-09-16T13:28:33.564583+0000","last_peered":"2024-09-16T13:28:33.564583+0000","last_clean":"2024-09-16T13:28:33.564583+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:28:33.564583+0000","last_undegraded":"2024-09-16T13:28:33.564583+0000","last_fullsized":"2024-09-16T13:28:33.564583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056234500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":51,"num_write":5,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.1a","version":"38'20","reported_seq":305,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.147498+0000","last_change":"2024-09-16T13:28:28.979457+0000","last_active":"2024-09-16T13:28:34.147498+0000","last_peered":"2024-09-16T13:28:34.147498+0000","last_clean":"2024-09-16T13:28:34.147498+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:28:34.147498+0000","last_undegraded":"2024-09-16T13:28:34.147498+0000","last_fullsized":"2024-09-16T13:28:34.147498+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058420599999999998,"stat_sum":{"num_bytes":10241930,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":272,"num_read_kb":2304,"num_write":18,"num_write_kb":314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.19","version":"38'14","reported_seq":35,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.999970+0000","last_change":"2024-09-16T13:28:28.983849+0000","last_active":"2024-09-16T13:28:33.999970+0000","last_peered":"2024-09-16T13:28:33.999970+0000","last_clean":"2024-09-16T13:28:33.999970+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:28:33.999970+0000","last_undegraded":"2024-09-16T13:28:33.999970+0000","last_fullsized":"2024-09-16T13:28:33.999970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041077799999999999,"stat_sum":{"num_bytes":4603378,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":12,"num_read_kb":0,"num_write":14,"num_write_kb":175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.18","version":"38'37","reported_seq":217,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.111440+0000","last_change":"2024-09-16T13:28:28.979247+0000","last_active":"2024-09-16T13:28:34.111440+0000","last_peered":"2024-09-16T13:28:34.111440+0000","last_clean":"2024-09-16T13:28:34.111440+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:28:34.111440+0000","last_undegraded":"2024-09-16T13:28:34.111440+0000","last_fullsized":"2024-09-16T13:28:34.111440+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039287600000000001,"stat_sum":{"num_bytes":8314014,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":417,"num_read_kb":417,"num_write":37,"num_write_kb":548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.17","version":"37'1","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.584343+0000","last_change":"2024-09-16T13:28:28.984991+0000","last_active":"2024-09-16T13:28:33.584343+0000","last_peered":"2024-09-16T13:28:33.584343+0000","last_clean":"2024-09-16T13:28:33.584343+0000","last_became_active":"2024-09-16T13:28:26.988783+0000","last_became_peered":"2024-09-16T13:28:26.988783+0000","last_unstale":"2024-09-16T13:28:33.584343+0000","last_undegraded":"2024-09-16T13:28:33.584343+0000","last_fullsized":"2024-09-16T13:28:33.584343+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:45:09.440727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080754000000000004,"stat_sum":{"num_bytes":425984,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.16","version":"35'2","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.184714+0000","last_change":"2024-09-16T13:28:30.062793+0000","last_active":"2024-09-16T13:28:32.184714+0000","last_peered":"2024-09-16T13:28:32.184714+0000","last_clean":"2024-09-16T13:28:32.184714+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:28:32.184714+0000","last_undegraded":"2024-09-16T13:28:32.184714+0000","last_fullsized":"2024-09-16T13:28:32.184714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039817100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.15","version":"38'9","reported_seq":26,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.067858+0000","last_change":"2024-09-16T13:28:28.991470+0000","last_active":"2024-09-16T13:28:34.067858+0000","last_peered":"2024-09-16T13:28:34.067858+0000","last_clean":"2024-09-16T13:28:34.067858+0000","last_became_active":"2024-09-16T13:28:26.987781+0000","last_became_peered":"2024-09-16T13:28:26.987781+0000","last_unstale":"2024-09-16T13:28:34.067858+0000","last_undegraded":"2024-09-16T13:28:34.067858+0000","last_fullsized":"2024-09-16T13:28:34.067858+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:47.563914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054,"stat_sum":{"num_bytes":425984,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.14","version":"38'43","reported_seq":57,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.144588+0000","last_change":"2024-09-16T13:28:28.984368+0000","last_active":"2024-09-16T13:28:34.144588+0000","last_peered":"2024-09-16T13:28:34.144588+0000","last_clean":"2024-09-16T13:28:34.144588+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:28:34.144588+0000","last_undegraded":"2024-09-16T13:28:34.144588+0000","last_fullsized":"2024-09-16T13:28:34.144588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000271229,"stat_sum":{"num_bytes":1540218,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":123,"num_read_kb":94,"num_write":67,"num_write_kb":1124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.13","version":"37'7","reported_seq":144,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.152594+0000","last_change":"2024-09-16T13:28:28.979874+0000","last_active":"2024-09-16T13:28:34.152594+0000","last_peered":"2024-09-16T13:28:34.152594+0000","last_clean":"2024-09-16T13:28:34.152594+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:28:34.152594+0000","last_undegraded":"2024-09-16T13:28:34.152594+0000","last_fullsized":"2024-09-16T13:28:34.152594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000816496,"stat_sum":{"num_bytes":9458245,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":125,"num_read_kb":2749,"num_write":7,"num_write_kb":99,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.12","version":"37'20","reported_seq":153,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.155918+0000","last_change":"2024-09-16T13:28:28.979687+0000","last_active":"2024-09-16T13:28:33.155918+0000","last_peered":"2024-09-16T13:28:33.155918+0000","last_clean":"2024-09-16T13:28:33.155918+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:28:33.155918+0000","last_undegraded":"2024-09-16T13:28:33.155918+0000","last_fullsized":"2024-09-16T13:28:33.155918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035537999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":356,"num_read_kb":303,"num_write":20,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.11","version":"38'82","reported_seq":194,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.151609+0000","last_change":"2024-09-16T13:28:28.983674+0000","last_active":"2024-09-16T13:28:34.151609+0000","last_peered":"2024-09-16T13:28:34.151609+0000","last_clean":"2024-09-16T13:28:34.151609+0000","last_became_active":"2024-09-16T13:28:26.988441+0000","last_became_peered":"2024-09-16T13:28:26.988441+0000","last_unstale":"2024-09-16T13:28:34.151609+0000","last_undegraded":"2024-09-16T13:28:34.151609+0000","last_fullsized":"2024-09-16T13:28:34.151609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:34:17.380757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035811000000000003,"stat_sum":{"num_bytes":2298199,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":4,"num_read":400,"num_read_kb":424,"num_write":118,"num_write_kb":539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.10","version":"38'8","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.127819+0000","last_change":"2024-09-16T13:28:28.983418+0000","last_active":"2024-09-16T13:28:34.127819+0000","last_peered":"2024-09-16T13:28:34.127819+0000","last_clean":"2024-09-16T13:28:34.127819+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:28:34.127819+0000","last_undegraded":"2024-09-16T13:28:34.127819+0000","last_fullsized":"2024-09-16T13:28:34.127819+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044577700000000001,"stat_sum":{"num_bytes":557056,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.002167+0000","last_change":"2024-09-16T13:28:12.801426+0000","last_active":"2024-09-16T13:28:32.002167+0000","last_peered":"2024-09-16T13:28:32.002167+0000","last_clean":"2024-09-16T13:28:32.002167+0000","last_became_active":"2024-09-16T13:28:10.312074+0000","last_became_peered":"2024-09-16T13:28:10.312074+0000","last_unstale":"2024-09-16T13:28:32.002167+0000","last_undegraded":"2024-09-16T13:28:32.002167+0000","last_fullsized":"2024-09-16T13:28:32.002167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:35.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"38'16","reported_seq":32,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.106828+0000","last_change":"2024-09-16T13:28:28.979658+0000","last_active":"2024-09-16T13:28:34.106828+0000","last_peered":"2024-09-16T13:28:34.106828+0000","last_clean":"2024-09-16T13:28:34.106828+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:28:34.106828+0000","last_undegraded":"2024-09-16T13:28:34.106828+0000","last_fullsized":"2024-09-16T13:28:34.106828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066082900000000004,"stat_sum":{"num_bytes":847324,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":3,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.024562+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:32.024562+0000","last_peered":"2024-09-16T13:28:32.024562+0000","last_clean":"2024-09-16T13:28:32.024562+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:28:32.024562+0000","last_undegraded":"2024-09-16T13:28:32.024562+0000","last_fullsized":"2024-09-16T13:28:32.024562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"38'77","reported_seq":99,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.082638+0000","last_change":"2024-09-16T13:28:28.983915+0000","last_active":"2024-09-16T13:28:34.082638+0000","last_peered":"2024-09-16T13:28:34.082638+0000","last_clean":"2024-09-16T13:28:34.082638+0000","last_became_active":"2024-09-16T13:28:26.988613+0000","last_became_peered":"2024-09-16T13:28:26.988613+0000","last_unstale":"2024-09-16T13:28:34.082638+0000","last_undegraded":"2024-09-16T13:28:34.082638+0000","last_fullsized":"2024-09-16T13:28:34.082638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:12:28.717624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043631399999999998,"stat_sum":{"num_bytes":1104243,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":380,"num_read_kb":303,"num_write":134,"num_write_kb":562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.002080+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:28:32.002080+0000","last_peered":"2024-09-16T13:28:32.002080+0000","last_clean":"2024-09-16T13:28:32.002080+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:28:32.002080+0000","last_undegraded":"2024-09-16T13:28:32.002080+0000","last_fullsized":"2024-09-16T13:28:32.002080+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"37'6","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.004011+0000","last_change":"2024-09-16T13:28:30.062555+0000","last_active":"2024-09-16T13:28:33.004011+0000","last_peered":"2024-09-16T13:28:33.004011+0000","last_clean":"2024-09-16T13:28:33.004011+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:28:33.004011+0000","last_undegraded":"2024-09-16T13:28:33.004011+0000","last_fullsized":"2024-09-16T13:28:33.004011+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000230199,"stat_sum":{"num_bytes":3948436,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":5,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.024454+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:28:32.024454+0000","last_peered":"2024-09-16T13:28:32.024454+0000","last_clean":"2024-09-16T13:28:32.024454+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:28:32.024454+0000","last_undegraded":"2024-09-16T13:28:32.024454+0000","last_fullsized":"2024-09-16T13:28:32.024454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"37'4","reported_seq":23,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.997953+0000","last_change":"2024-09-16T13:28:28.979887+0000","last_active":"2024-09-16T13:28:32.997953+0000","last_peered":"2024-09-16T13:28:32.997953+0000","last_clean":"2024-09-16T13:28:32.997953+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:28:32.997953+0000","last_undegraded":"2024-09-16T13:28:32.997953+0000","last_fullsized":"2024-09-16T13:28:32.997953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053138000000000003,"stat_sum":{"num_bytes":1977936,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":8,"num_read_kb":2,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.995957+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:32.995957+0000","last_peered":"2024-09-16T13:28:32.995957+0000","last_clean":"2024-09-16T13:28:32.995957+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:32.995957+0000","last_undegraded":"2024-09-16T13:28:32.995957+0000","last_fullsized":"2024-09-16T13:28:32.995957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":32,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.061200+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:33.061200+0000","last_peered":"2024-09-16T13:28:33.061200+0000","last_clean":"2024-09-16T13:28:33.061200+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:33.061200+0000","last_undegraded":"2024-09-16T13:28:33.061200+0000","last_fullsized":"2024-09-16T13:28:33.061200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"37'11","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.997565+0000","last_change":"2024-09-16T13:28:28.979855+0000","last_active":"2024-09-16T13:28:32.997565+0000","last_peered":"2024-09-16T13:28:32.997565+0000","last_clean":"2024-09-16T13:28:32.997565+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:28:32.997565+0000","last_undegraded":"2024-09-16T13:28:32.997565+0000","last_fullsized":"2024-09-16T13:28:32.997565+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039579799999999999,"stat_sum":{"num_bytes":3670016,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":10,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.023691+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:28:32.023691+0000","last_peered":"2024-09-16T13:28:32.023691+0000","last_clean":"2024-09-16T13:28:32.023691+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:28:32.023691+0000","last_undegraded":"2024-09-16T13:28:32.023691+0000","last_fullsized":"2024-09-16T13:28:32.023691+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:31.995466+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:28:31.995466+0000","last_peered":"2024-09-16T13:28:31.995466+0000","last_clean":"2024-09-16T13:28:31.995466+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:28:31.995466+0000","last_undegraded":"2024-09-16T13:28:31.995466+0000","last_fullsized":"2024-09-16T13:28:31.995466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"36'2","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.305625+0000","last_change":"2024-09-16T13:28:28.980209+0000","last_active":"2024-09-16T13:28:32.305625+0000","last_peered":"2024-09-16T13:28:32.305625+0000","last_clean":"2024-09-16T13:28:32.305625+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:28:32.305625+0000","last_undegraded":"2024-09-16T13:28:32.305625+0000","last_fullsized":"2024-09-16T13:28:32.305625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069467499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.184443+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:28:32.184443+0000","last_peered":"2024-09-16T13:28:32.184443+0000","last_clean":"2024-09-16T13:28:32.184443+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:28:32.184443+0000","last_undegraded":"2024-09-16T13:28:32.184443+0000","last_fullsized":"2024-09-16T13:28:32.184443+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:31.394402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.005867+0000","last_change":"2024-09-16T13:28:12.959240+0000","last_active":"2024-09-16T13:28:33.005867+0000","last_peered":"2024-09-16T13:28:33.005867+0000","last_clean":"2024-09-16T13:28:33.005867+0000","last_became_active":"2024-09-16T13:28:10.312127+0000","last_became_peered":"2024-09-16T13:28:10.312127+0000","last_unstale":"2024-09-16T13:28:33.005867+0000","last_undegraded":"2024-09-16T13:28:33.005867+0000","last_fullsized":"2024-09-16T13:28:33.005867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:22.742224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030101900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"36'3","reported_seq":19,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.378705+0000","last_change":"2024-09-16T13:28:30.062973+0000","last_active":"2024-09-16T13:28:32.378705+0000","last_peered":"2024-09-16T13:28:32.378705+0000","last_clean":"2024-09-16T13:28:32.378705+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:28:32.378705+0000","last_undegraded":"2024-09-16T13:28:32.378705+0000","last_fullsized":"2024-09-16T13:28:32.378705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049121500000000003,"stat_sum":{"num_bytes":2771150,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.001460+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:33.001460+0000","last_peered":"2024-09-16T13:28:33.001460+0000","last_clean":"2024-09-16T13:28:33.001460+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:33.001460+0000","last_undegraded":"2024-09-16T13:28:33.001460+0000","last_fullsized":"2024-09-16T13:28:33.001460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:52.853771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":31,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.061179+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:33.061179+0000","last_peered":"2024-09-16T13:28:33.061179+0000","last_clean":"2024-09-16T13:28:33.061179+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:33.061179+0000","last_undegraded":"2024-09-16T13:28:33.061179+0000","last_fullsized":"2024-09-16T13:28:33.061179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":118,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.005907+0000","last_change":"2024-09-16T13:28:05.835186+0000","last_active":"2024-09-16T13:28:33.005907+0000","last_peered":"2024-09-16T13:28:33.005907+0000","last_clean":"2024-09-16T13:28:33.005907+0000","last_became_active":"2024-09-16T13:28:05.834235+0000","last_became_peered":"2024-09-16T13:28:05.834235+0000","last_unstale":"2024-09-16T13:28:33.005907+0000","last_undegraded":"2024-09-16T13:28:33.005907+0000","last_fullsized":"2024-09-16T13:28:33.005907+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:57.676141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"36'9","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.343577+0000","last_change":"2024-09-16T13:28:28.989279+0000","last_active":"2024-09-16T13:28:32.343577+0000","last_peered":"2024-09-16T13:28:32.343577+0000","last_clean":"2024-09-16T13:28:32.343577+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:28:32.343577+0000","last_undegraded":"2024-09-16T13:28:32.343577+0000","last_fullsized":"2024-09-16T13:28:32.343577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024121900000000001,"stat_sum":{"num_bytes":4823606,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":0,"num_write":9,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.8","version":"38'20","reported_seq":112,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.129587+0000","last_change":"2024-09-16T13:28:28.991427+0000","last_active":"2024-09-16T13:28:34.129587+0000","last_peered":"2024-09-16T13:28:34.129587+0000","last_clean":"2024-09-16T13:28:34.129587+0000","last_became_active":"2024-09-16T13:28:26.988593+0000","last_became_peered":"2024-09-16T13:28:26.988593+0000","last_unstale":"2024-09-16T13:28:34.129587+0000","last_undegraded":"2024-09-16T13:28:34.129587+0000","last_fullsized":"2024-09-16T13:28:34.129587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:32.662488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024448900000000002,"stat_sum":{"num_bytes":4352518,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":86,"num_read_kb":468,"num_write":16,"num_write_kb":1038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.9","version":"38'21","reported_seq":123,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.152651+0000","last_change":"2024-09-16T13:28:28.979223+0000","last_active":"2024-09-16T13:28:34.152651+0000","last_peered":"2024-09-16T13:28:34.152651+0000","last_clean":"2024-09-16T13:28:34.152651+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:28:34.152651+0000","last_undegraded":"2024-09-16T13:28:34.152651+0000","last_fullsized":"2024-09-16T13:28:34.152651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039311499999999998,"stat_sum":{"num_bytes":9920646,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":101,"num_read_kb":1624,"num_write":18,"num_write_kb":718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.a","version":"37'7","reported_seq":23,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.000317+0000","last_change":"2024-09-16T13:28:28.990059+0000","last_active":"2024-09-16T13:28:33.000317+0000","last_peered":"2024-09-16T13:28:33.000317+0000","last_clean":"2024-09-16T13:28:33.000317+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:28:33.000317+0000","last_undegraded":"2024-09-16T13:28:33.000317+0000","last_fullsized":"2024-09-16T13:28:33.000317+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086978299999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.b","version":"36'7","reported_seq":45,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T13:28:32.343152+0000","last_change":"2024-09-16T13:28:30.062577+0000","last_active":"2024-09-16T13:28:32.343152+0000","last_peered":"2024-09-16T13:28:32.343152+0000","last_clean":"2024-09-16T13:28:32.343152+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:28:32.343152+0000","last_undegraded":"2024-09-16T13:28:32.343152+0000","last_fullsized":"2024-09-16T13:28:32.343152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029103499999999998,"stat_sum":{"num_bytes":3585061,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":131,"num_write":7,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.c","version":"38'114","reported_seq":387,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.097992+0000","last_change":"2024-09-16T13:28:28.979711+0000","last_active":"2024-09-16T13:28:34.097992+0000","last_peered":"2024-09-16T13:28:34.097992+0000","last_clean":"2024-09-16T13:28:34.097992+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:28:34.097992+0000","last_undegraded":"2024-09-16T13:28:34.097992+0000","last_fullsized":"2024-09-16T13:28:34.097992+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071623100000000001,"stat_sum":{"num_bytes":122,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":1151,"num_read_kb":979,"num_write":165,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.d","version":"38'23","reported_seq":115,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.138399+0000","last_change":"2024-09-16T13:28:28.979910+0000","last_active":"2024-09-16T13:28:34.138399+0000","last_peered":"2024-09-16T13:28:34.138399+0000","last_clean":"2024-09-16T13:28:34.138399+0000","last_became_active":"2024-09-16T13:28:26.987044+0000","last_became_peered":"2024-09-16T13:28:26.987044+0000","last_unstale":"2024-09-16T13:28:34.138399+0000","last_undegraded":"2024-09-16T13:28:34.138399+0000","last_fullsized":"2024-09-16T13:28:34.138399+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:11.809480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084175000000000005,"stat_sum":{"num_bytes":9117252,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":86,"num_read_kb":1832,"num_write":19,"num_write_kb":1100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.e","version":"38'14","reported_seq":31,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-16T13:28:34.099388+0000","last_change":"2024-09-16T13:28:28.980116+0000","last_active":"2024-09-16T13:28:34.099388+0000","last_peered":"2024-09-16T13:28:34.099388+0000","last_clean":"2024-09-16T13:28:34.099388+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:28:34.099388+0000","last_undegraded":"2024-09-16T13:28:34.099388+0000","last_fullsized":"2024-09-16T13:28:34.099388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010142790000000001,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":6,"num_read_kb":0,"num_write":11,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.f","version":"37'11","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T13:28:33.061650+0000","last_change":"2024-09-16T13:28:28.980140+0000","last_active":"2024-09-16T13:28:33.061650+0000","last_peered":"2024-09-16T13:28:33.061650+0000","last_clean":"2024-09-16T13:28:33.061650+0000","last_became_active":"2024-09-16T13:28:26.986290+0000","last_became_peered":"2024-09-16T13:28:26.986290+0000","last_unstale":"2024-09-16T13:28:33.061650+0000","last_undegraded":"2024-09-16T13:28:33.061650+0000","last_fullsized":"2024-09-16T13:28:33.061650+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:51.239301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074187000000000005,"stat_sum":{"num_bytes":4463698,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":4,"num_read_kb":0,"num_write":9,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":97938836,"num_objects":128,"num_object_clones":31,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":33,"num_read":3640,"num_read_kb":11831,"num_write":765,"num_write_kb":9707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3092480,"data_stored":14037130,"data_compressed":1254215,"data_compressed_allocated":2199552,"data_compressed_original":13594624,"omap_allocated":0,"internal_metadata":0},"log_size":618,"ondisk_log_size":618,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11198,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302108,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069732,"statfs":{"total":96636764160,"available":96327405568,"internally_reserved":0,"allocated":606208,"data_stored":1058173,"data_compressed":44185,"data_compressed_allocated":106496,"data_compressed_original":839680,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297900,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073940,"statfs":{"total":96636764160,"available":96331714560,"internally_reserved":0,"allocated":552960,"data_stored":481696,"data_compressed":54700,"data_compressed_allocated":69632,"data_compressed_original":270336,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509451,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302440,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069400,"statfs":{"total":96636764160,"available":96327065600,"internally_reserved":0,"allocated":950272,"data_stored":951534,"data_compressed":277732,"data_compressed_allocated":331776,"data_compressed_original":675840,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302116,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069724,"statfs":{"total":96636764160,"available":96327397376,"internally_reserved":0,"allocated":618496,"data_stored":482006,"data_compressed":119627,"data_compressed_allocated":122880,"data_compressed_original":274432,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297944,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073896,"statfs":{"total":96636764160,"available":96331669504,"internally_reserved":0,"allocated":606208,"data_stored":515874,"data_compressed":92689,"data_compressed_allocated":118784,"data_compressed_original":311296,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073916,"statfs":{"total":96636764160,"available":96331689984,"internally_reserved":0,"allocated":581632,"data_stored":429957,"data_compressed":60166,"data_compressed_allocated":77824,"data_compressed_original":204800,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298192,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073648,"statfs":{"total":96636764160,"available":96331415552,"internally_reserved":0,"allocated":856064,"data_stored":776125,"data_compressed":276877,"data_compressed_allocated":299008,"data_compressed_original":524288,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297912,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073928,"statfs":{"total":96636764160,"available":96331702272,"internally_reserved":0,"allocated":565248,"data_stored":902488,"data_compressed":65225,"data_compressed_allocated":110592,"data_compressed_original":700416,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.188}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":165762,"data_compressed":73134,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":606336,"data_compressed":276877,"data_compressed_allocated":299008,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":260187,"data_compressed":60166,"data_compressed_allocated":77824,"data_compressed_original":204800,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":675840,"data_stored":4483611,"data_compressed":245490,"data_compressed_allocated":552960,"data_compressed_original":4435968,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":815104,"data_stored":3733439,"data_compressed":347554,"data_compressed_allocated":589824,"data_compressed_original":3624960,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":1896316,"data_compressed":85195,"data_compressed_allocated":217088,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":352689,"data_compressed":92689,"data_compressed_allocated":118784,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":2538790,"data_compressed":73110,"data_compressed_allocated":270336,"data_compressed_original":2510848,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:28:38.729 INFO:tasks.thrashosds.thrasher:Setting 4.18 to [1, 2, 7, 6] 2024-09-16T13:28:38.729 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.18', '1', '2', '7', '6'] 2024-09-16T13:28:38.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.18 1 2 7 6 2024-09-16T13:28:38.750 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:38.749+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:38.759 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.759 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:38.759 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:38.759 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.768 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:38.828 INFO:teuthology.orchestra.run.smithi076.stdout:563 trunc from 0xbb5420c to 0xdc0fbfd 2024-09-16T13:28:38.835 INFO:teuthology.orchestra.run.smithi076.stdout:564 read 0x8617d21 thru 0x8623f6c (0xc24c bytes) 2024-09-16T13:28:38.835 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.835 INFO:teuthology.orchestra.run.smithi076.stdout:566 writesame 0x75a9dac thru 0x75aa1a2 data_size 0x7(0x3f7 bytes) 2024-09-16T13:28:38.838 INFO:teuthology.orchestra.run.smithi076.stdout:567 write 0x3410eae thru 0x3415104 (0x4257 bytes) 2024-09-16T13:28:38.841 INFO:teuthology.orchestra.run.smithi076.stdout:568 punch from 0x721b687 to 0x7222a96, (0x740f bytes) 2024-09-16T13:28:38.842 INFO:teuthology.orchestra.run.smithi076.stdout:569 punch from 0xcc69d86 to 0xcc71bae, (0x7e28 bytes) 2024-09-16T13:28:38.842 INFO:teuthology.orchestra.run.smithi076.stdout:570 writesame 0x5346a03 thru 0x5356132 data_size 0x1c(0xf730 bytes) 2024-09-16T13:28:38.851 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:38.849+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:38.853 INFO:teuthology.orchestra.run.smithi076.stdout:571 trunc from 0xdc0fbfd to 0x7f1c4af 2024-09-16T13:28:38.862 INFO:teuthology.orchestra.run.smithi076.stdout:572 read 0x2dbdfb7 thru 0x2dc4f3f (0x6f89 bytes) 2024-09-16T13:28:38.863 INFO:teuthology.orchestra.run.smithi076.stdout:573 trunc from 0x7f1c4af to 0x79f352d 2024-09-16T13:28:38.868 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.868 INFO:teuthology.orchestra.run.smithi076.stdout:578 punch from 0x22423e1 to 0x2249372, (0x6f91 bytes) 2024-09-16T13:28:38.869 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.876 INFO:teuthology.orchestra.run.smithi076.stdout:582 writesame 0x96b3921 thru 0x96bade7 data_size 0x5(0x74c7 bytes) 2024-09-16T13:28:38.876 INFO:teuthology.orchestra.run.smithi076.stdout:583 read 0x70f0123 thru 0x70f4f92 (0x4e70 bytes) 2024-09-16T13:28:38.876 INFO:teuthology.orchestra.run.smithi076.stdout:584 write 0x1170096 thru 0x117903e (0x8fa9 bytes) 2024-09-16T13:28:38.887 INFO:teuthology.orchestra.run.smithi076.stdout:586 writesame 0xabbf7dc thru 0xabc2efe data_size 0x5(0x3723 bytes) 2024-09-16T13:28:38.888 INFO:teuthology.orchestra.run.smithi076.stdout:587 write 0x7dc9c9 thru 0x7e06be (0x3cf6 bytes) 2024-09-16T13:28:38.889 INFO:teuthology.orchestra.run.smithi076.stdout:588 punch from 0x2e1322 to 0x2ee004, (0xcce2 bytes) 2024-09-16T13:28:38.889 INFO:teuthology.orchestra.run.smithi076.stdout:589 punch from 0x49090d5 to 0x4910cc9, (0x7bf4 bytes) 2024-09-16T13:28:38.890 INFO:teuthology.orchestra.run.smithi076.stdout:590 writesame 0x56cc265 thru 0x56ce0eb data_size 0x5(0x1e87 bytes) 2024-09-16T13:28:38.902 INFO:teuthology.orchestra.run.smithi076.stdout:593 trunc from 0xabc2eff to 0x220ad2d 2024-09-16T13:28:38.911 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:38.911 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:38.911 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:38.911 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:38.912 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:38.913 INFO:teuthology.orchestra.run.smithi076.stdout:594 punch from 0x1c16a20 to 0x1c1a58a, (0x3b6a bytes) 2024-09-16T13:28:38.913 INFO:teuthology.orchestra.run.smithi076.stdout:595 punch from 0x1d93b0 to 0x1dde18, (0x4a68 bytes) 2024-09-16T13:28:38.914 INFO:teuthology.orchestra.run.smithi076.stdout:597 write 0x964620 thru 0x96b3a0 (0x6d81 bytes) 2024-09-16T13:28:38.916 INFO:teuthology.orchestra.run.smithi076.stdout:599 write 0x1649004 thru 0x164f594 (0x6591 bytes) 2024-09-16T13:28:38.919 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xe73ef8b 2024-09-16T13:28:38.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:38.934 INFO:teuthology.orchestra.run.smithi076.stdout:600 trunc from 0x220ad2d to 0xe73ef8b 2024-09-16T13:28:38.940 INFO:teuthology.orchestra.run.smithi076.stdout:601 read 0xbf28ced thru 0xbf29d13 (0x1027 bytes) 2024-09-16T13:28:38.940 INFO:teuthology.orchestra.run.smithi076.stdout:602 punch from 0xb1a104b to 0xb1af038, (0xdfed bytes) 2024-09-16T13:28:38.940 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.940 INFO:teuthology.orchestra.run.smithi076.stdout:604 writesame 0xcc85167 thru 0xcc8a1bd data_size 0x1(0x5057 bytes) 2024-09-16T13:28:38.966 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:38.964+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:38.975 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:38.975 INFO:teuthology.orchestra.run.smithi076.stdout:607 punch from 0x25a89f6 to 0x25b085c, (0x7e66 bytes) 2024-09-16T13:28:38.975 INFO:teuthology.orchestra.run.smithi076.stdout:608 read 0x8c6b7c2 thru 0x8c75d74 (0xa5b3 bytes) 2024-09-16T13:28:38.978 INFO:teuthology.orchestra.run.smithi076.stdout:609 trunc from 0xe73ef8b to 0xbdd400b 2024-09-16T13:28:38.999 INFO:teuthology.orchestra.run.smithi076.stdout:610 trunc from 0xbdd400b to 0x31ce631 2024-09-16T13:28:39.005 INFO:teuthology.orchestra.run.smithi076.stdout:611 read 0x2b4b392 thru 0x2b4d5e1 (0x2250 bytes) 2024-09-16T13:28:39.005 INFO:teuthology.orchestra.run.smithi076.stdout:614 read 0x2cd7e4 thru 0x2cddcd (0x5ea bytes) 2024-09-16T13:28:39.021 INFO:teuthology.orchestra.run.smithi076.stdout:615 writesame 0xa140160 thru 0xa145a4b data_size 0x2a(0x58ec bytes) 2024-09-16T13:28:39.021 INFO:teuthology.orchestra.run.smithi076.stdout:616 read 0x72fb05b thru 0x7309196 (0xe13c bytes) 2024-09-16T13:28:39.021 INFO:teuthology.orchestra.run.smithi076.stdout:619 writesame 0x7950675 thru 0x795c58c data_size 0x14(0xbf18 bytes) 2024-09-16T13:28:39.024 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:39.024 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:39.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.732264411000003, 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.025 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:28:39.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:28:39.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030914213999999, 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.037 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030791817999997, 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:28:39.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:28:39.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030679429999999, 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:28:39.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030616616000003, 2024-09-16T13:28:39.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:28:39.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030469185999998, 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.055 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:28:39.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:28:39.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030454003000003, 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:28:39.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:28:39.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.066 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030432955999999, 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:28:39.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030345476000001, 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:28:39.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:28:39.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030322966999996, 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.079 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.084 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030276639999997, 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:39.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030109883000002, 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.090 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.095 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.030095117999998, 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:28:39.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.097 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:39.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:28:39.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.684772801000001, 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.105 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:28:39.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.684735123000003, 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.111 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:39.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:28:39.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:28:39.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.116 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.684712310000002, 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.684627097000003, 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:39.127 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3780481719999997, 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:28:39.131 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3521399379999997, 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:39.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:28:39.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:28:39.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.135 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.2833747830000002, 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:39.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:28:39.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:39.140 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:39.142 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:39.140+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:39.143 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:39.144 INFO:teuthology.orchestra.run.smithi076.stdout:620 writesame 0x3f5bd08 thru 0x3f66f3e data_size 0x1(0xb237 bytes) 2024-09-16T13:28:39.144 INFO:teuthology.orchestra.run.smithi076.stdout:621 write 0xd9b3e54 thru 0xd9b653c (0x26e9 bytes) 2024-09-16T13:28:39.144 INFO:teuthology.orchestra.run.smithi076.stdout:623 write 0x5cd7cf3 thru 0x5ce3d84 (0xc092 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:625 punch from 0x94052fd to 0x94146b5, (0xf3b8 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:626 write 0x7a2966e thru 0x7a2e746 (0x50d9 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:629 write 0x1d1fe24 thru 0x1d2ddd4 (0xdfb1 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:630 trunc from 0xd9b653d to 0x7e58982 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:633 read 0x2912d39 thru 0x291bd3c (0x9004 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:636 writesame 0x1a1d797 thru 0x1a2bf0c data_size 0x56(0xe776 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:637 write 0x41bc2d1 thru 0x41bc680 (0x3b0 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:639 punch from 0x49bc2b4 to 0x49ca293, (0xdfdf bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:640 punch from 0x65a51f4 to 0x65a8cb6, (0x3ac2 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:641 write 0xa465b0f thru 0xa473765 (0xdc57 bytes) 2024-09-16T13:28:39.145 INFO:teuthology.orchestra.run.smithi076.stdout:642 write 0x9b373c0 thru 0x9b379d4 (0x615 bytes) 2024-09-16T13:28:39.146 INFO:teuthology.orchestra.run.smithi076.stdout:645 punch from 0x79c01f0 to 0x79ce022, (0xde32 bytes) 2024-09-16T13:28:39.146 INFO:teuthology.orchestra.run.smithi076.stdout:648 punch from 0x6cc5a98 to 0x6cc8163, (0x26cb bytes) 2024-09-16T13:28:39.146 INFO:teuthology.orchestra.run.smithi076.stdout:649 read 0x7b1cfbf thru 0x7b271a1 (0xa1e3 bytes) 2024-09-16T13:28:39.146 INFO:teuthology.orchestra.run.smithi076.stdout:650 writesame 0xb3f561c thru 0xb3f9b2a data_size 0x47(0x450f bytes) 2024-09-16T13:28:39.146 INFO:teuthology.orchestra.run.smithi076.stdout:651 writesame 0x9558c16 thru 0x9563194 data_size 0x1(0xa57f bytes) 2024-09-16T13:28:39.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:39.180 INFO:teuthology.orchestra.run.smithi076.stdout:652 writesame 0x3878b3c thru 0x38844cb data_size 0x8(0xb990 bytes) 2024-09-16T13:28:39.185 INFO:teuthology.orchestra.run.smithi076.stdout:653 writesame 0x7d8f309 thru 0x7d9a4ee data_size 0x7(0xb1e6 bytes) 2024-09-16T13:28:39.206 INFO:teuthology.orchestra.run.smithi076.stdout:654 trunc from 0xb3f9b2b to 0x35192d9 2024-09-16T13:28:39.230 INFO:teuthology.orchestra.run.smithi076.stdout:655 write 0x7a3873a thru 0x7a3a390 (0x1c57 bytes) 2024-09-16T13:28:39.242 INFO:teuthology.orchestra.run.smithi076.stdout:656 write 0x91bddaa thru 0x91c674c (0x89a3 bytes) 2024-09-16T13:28:39.242 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:39.240+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:39.257 INFO:teuthology.orchestra.run.smithi076.stdout:658 trunc from 0x91c674d to 0x2b38fd9 2024-09-16T13:28:39.258 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:39.258 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:39.258 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:39.259 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:39.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:39.313 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:39.313 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:39.331 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:39.344 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:39.342+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:39.348 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:39.349 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:39.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:39.364 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:39.392 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:39.393 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:39.409 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:39.423 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:39.423 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:39.429 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:39.430 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:39.432 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:39.446 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:39.447 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:39.445+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:39.482 INFO:teuthology.orchestra.run.smithi076.stdout:659 write 0x6ebc56a thru 0x6ec4439 (0x7ed0 bytes) 2024-09-16T13:28:39.489 INFO:teuthology.orchestra.run.smithi076.stdout:660 write 0x5d033bc thru 0x5d11ce1 (0xe926 bytes) 2024-09-16T13:28:39.494 INFO:teuthology.orchestra.run.smithi076.stdout:661 read 0x6d28a53 thru 0x6d2bb24 (0x30d2 bytes) 2024-09-16T13:28:39.494 INFO:teuthology.orchestra.run.smithi076.stdout:662 read 0xcd99b9 thru 0xce20fd (0x8745 bytes) 2024-09-16T13:28:39.504 INFO:teuthology.orchestra.run.smithi076.stdout:663 writesame 0xaf716df thru 0xaf785c6 data_size 0x9c(0x6ee8 bytes) 2024-09-16T13:28:39.505 INFO:teuthology.orchestra.run.smithi076.stdout:664 write 0x4d8a0a thru 0x4db452 (0x2a49 bytes) 2024-09-16T13:28:39.518 INFO:teuthology.orchestra.run.smithi076.stdout:665 trunc from 0xaf785c7 to 0x672d19 2024-09-16T13:28:39.545 INFO:teuthology.orchestra.run.smithi076.stdout:670 trunc from 0x672d19 to 0x96e275e 2024-09-16T13:28:39.545 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:39.543+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:39.549 INFO:teuthology.orchestra.run.smithi076.stdout:674 writesame 0x732247f thru 0x732731c data_size 0x1d(0x4e9e bytes) 2024-09-16T13:28:39.552 INFO:teuthology.orchestra.run.smithi076.stdout:675 punch from 0x6799d05 to 0x67a4a39, (0xad34 bytes) 2024-09-16T13:28:39.552 INFO:teuthology.orchestra.run.smithi076.stdout:677 read 0x2a8bbf6 thru 0x2a9b32c (0xf737 bytes) 2024-09-16T13:28:39.562 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:39.562 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:39.564 INFO:teuthology.orchestra.run.smithi076.stdout:678 trunc from 0x96e275e to 0x10b3cb5 2024-09-16T13:28:39.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:39.573 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.588 INFO:teuthology.orchestra.run.smithi076.stdout:680 write 0x9b52168 thru 0x9b531f0 (0x1089 bytes) 2024-09-16T13:28:39.591 INFO:teuthology.orchestra.run.smithi076.stdout:681 writesame 0x95fe6df thru 0x9604895 data_size 0x1(0x61b7 bytes) 2024-09-16T13:28:39.627 INFO:teuthology.orchestra.run.smithi076.stdout:682 write 0x398906 thru 0x39ac02 (0x22fd bytes) 2024-09-16T13:28:39.629 INFO:teuthology.orchestra.run.smithi076.stdout:683 read 0x8e9c28e thru 0x8ea58e5 (0x9658 bytes) 2024-09-16T13:28:39.629 INFO:teuthology.orchestra.run.smithi076.stdout:684 read 0x9b147e1 thru 0x9b19251 (0x4a71 bytes) 2024-09-16T13:28:39.640 INFO:teuthology.orchestra.run.smithi076.stdout:685 write 0xea8f312 thru 0xea937a4 (0x4493 bytes) 2024-09-16T13:28:39.643 INFO:teuthology.orchestra.run.smithi076.stdout:687 write 0xe5ebff2 thru 0xe5ef740 (0x374f bytes) 2024-09-16T13:28:39.646 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.646 INFO:teuthology.orchestra.run.smithi076.stdout:689 write 0x2843cf0 thru 0x28528b0 (0xebc1 bytes) 2024-09-16T13:28:39.646 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:39.644+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:39.656 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.659 INFO:teuthology.orchestra.run.smithi076.stdout:691 trunc from 0xea937a5 to 0xc75ad0c 2024-09-16T13:28:39.669 INFO:teuthology.orchestra.run.smithi076.stdout:694 write 0x10b42f2 thru 0x10bd63f (0x934e bytes) 2024-09-16T13:28:39.673 INFO:teuthology.orchestra.run.smithi076.stdout:695 punch from 0x6032e81 to 0x6035da6, (0x2f25 bytes) 2024-09-16T13:28:39.674 INFO:teuthology.orchestra.run.smithi076.stdout:696 trunc from 0xc75ad0c to 0xbad274c 2024-09-16T13:28:39.683 INFO:teuthology.orchestra.run.smithi076.stdout:698 trunc from 0xbad274c to 0x8496f3f 2024-09-16T13:28:39.684 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:39.684 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:39.693 INFO:teuthology.orchestra.run.smithi076.stdout:702 read 0x145fd9 thru 0x15365c (0xd684 bytes) 2024-09-16T13:28:39.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:39.694 INFO:teuthology.orchestra.run.smithi076.stdout:703 punch from 0x66a1416 to 0x66b12ba, (0xfea4 bytes) 2024-09-16T13:28:39.694 INFO:teuthology.orchestra.run.smithi076.stdout:704 punch from 0x8b9efb to 0x8c6df5, (0xcefa bytes) 2024-09-16T13:28:39.704 INFO:teuthology.orchestra.run.smithi076.stdout:705 write 0xc8fb4d1 thru 0xc906b88 (0xb6b8 bytes) 2024-09-16T13:28:39.707 INFO:teuthology.orchestra.run.smithi076.stdout:706 punch from 0x5ccce2b to 0x5cd7519, (0xa6ee bytes) 2024-09-16T13:28:39.715 INFO:teuthology.orchestra.run.smithi076.stdout:707 writesame 0xe80a9db thru 0xe80ab1a data_size 0xa(0x140 bytes) 2024-09-16T13:28:39.715 INFO:teuthology.orchestra.run.smithi076.stdout:710 write 0x2efd743 thru 0x2f01ced (0x45ab bytes) 2024-09-16T13:28:39.720 INFO:teuthology.orchestra.run.smithi076.stdout:711 write 0x40a15e6 thru 0x40a9b2f (0x854a bytes) 2024-09-16T13:28:39.724 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.734 INFO:teuthology.orchestra.run.smithi076.stdout:714 trunc from 0xe80ab1b to 0x6060c7c 2024-09-16T13:28:39.744 INFO:teuthology.orchestra.run.smithi076.stdout:715 punch from 0x5c438a4 to 0x5c47482, (0x3bde bytes) 2024-09-16T13:28:39.746 INFO:teuthology.orchestra.run.smithi076.stdout:716 trunc from 0x6060c7c to 0x86e955b 2024-09-16T13:28:39.747 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:39.745+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:39.752 INFO:teuthology.orchestra.run.smithi076.stdout:717 punch from 0x5df3a7f to 0x5e00459, (0xc9da bytes) 2024-09-16T13:28:39.762 INFO:teuthology.orchestra.run.smithi076.stdout:718 write 0xc8d6233 thru 0xc8db24b (0x5019 bytes) 2024-09-16T13:28:39.779 INFO:teuthology.orchestra.run.smithi076.stdout:719 trunc from 0xc8db24c to 0x1789c91 2024-09-16T13:28:39.790 INFO:teuthology.orchestra.run.smithi076.stdout:720 read 0xa71b06 thru 0xa79463 (0x795e bytes) 2024-09-16T13:28:39.802 INFO:teuthology.orchestra.run.smithi076.stdout:722 write 0x6e95ba4 thru 0x6e9a2e1 (0x473e bytes) 2024-09-16T13:28:39.805 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.815 INFO:teuthology.orchestra.run.smithi076.stdout:724 write 0xa6ca80c thru 0xa6d561d (0xae12 bytes) 2024-09-16T13:28:39.818 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:39.818 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:39.818 INFO:teuthology.orchestra.run.smithi076.stdout:725 read 0xcd44b4 thru 0xce02e3 (0xbe30 bytes) 2024-09-16T13:28:39.818 INFO:teuthology.orchestra.run.smithi076.stdout:726 writesame 0x9a510cb thru 0x9a5498c data_size 0x5(0x38c2 bytes) 2024-09-16T13:28:39.824 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.827 INFO:teuthology.orchestra.run.smithi076.stdout:728 trunc from 0xa6d561e to 0x775febd 2024-09-16T13:28:39.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:39.837 INFO:teuthology.orchestra.run.smithi076.stdout:730 write 0x6aa5267 thru 0x6aaa778 (0x5512 bytes) 2024-09-16T13:28:39.840 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.840 INFO:teuthology.orchestra.run.smithi076.stdout:732 write 0x2895991 thru 0x28a2c53 (0xd2c3 bytes) 2024-09-16T13:28:39.845 INFO:teuthology.orchestra.run.smithi076.stdout:733 write 0x380cbb4 thru 0x381b278 (0xe6c5 bytes) 2024-09-16T13:28:39.847 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:39.845+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:39.850 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:39.850 INFO:teuthology.orchestra.run.smithi076.stdout:737 writesame 0x62aacfa thru 0x62ac31a data_size 0xb(0x1621 bytes) 2024-09-16T13:28:39.865 INFO:teuthology.orchestra.run.smithi076.stdout:738 writesame 0xbf8c142 thru 0xbf92fcc data_size 0x1(0x6e8b bytes) 2024-09-16T13:28:39.865 INFO:teuthology.orchestra.run.smithi076.stdout:741 write 0xaca239f thru 0xacac082 (0x9ce4 bytes) 2024-09-16T13:28:39.872 INFO:teuthology.orchestra.run.smithi076.stdout:743 trunc from 0xbf92fcd to 0x963ed7e 2024-09-16T13:28:39.883 INFO:teuthology.orchestra.run.smithi076.stdout:744 trunc from 0x963ed7e to 0x90db1c1 2024-09-16T13:28:39.898 INFO:teuthology.orchestra.run.smithi076.stdout:747 write 0xdacec59 thru 0xdad9715 (0xaabd bytes) 2024-09-16T13:28:39.906 INFO:teuthology.orchestra.run.smithi076.stdout:748 write 0xe3c36e0 thru 0xe3c9d85 (0x66a6 bytes) 2024-09-16T13:28:39.910 INFO:teuthology.orchestra.run.smithi076.stdout:749 read 0x7716a97 thru 0x7717c4e (0x11b8 bytes) 2024-09-16T13:28:39.911 INFO:teuthology.orchestra.run.smithi076.stdout:750 write 0x2563da1 thru 0x256706f (0x32cf bytes) 2024-09-16T13:28:39.914 INFO:teuthology.orchestra.run.smithi076.stdout:751 read 0x9476c77 thru 0x947e2c6 (0x7650 bytes) 2024-09-16T13:28:39.914 INFO:teuthology.orchestra.run.smithi076.stdout:753 punch from 0x578f4e1 to 0x579600b, (0x6b2a bytes) 2024-09-16T13:28:39.914 INFO:teuthology.orchestra.run.smithi076.stdout:757 read 0x7415243 thru 0x741f33f (0xa0fd bytes) 2024-09-16T13:28:39.914 INFO:teuthology.orchestra.run.smithi076.stdout:758 clone 7 order 18 su 2048 sc 5 2024-09-16T13:28:39.943 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:39.944 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:39.948 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:39.946+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:39.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:40.023 INFO:teuthology.orchestra.run.smithi076.stderr:set 4.18 pg_upmap_items mapping to [1->2,7->6] 2024-09-16T13:28:40.049 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:40.047+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:40.103 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.103 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.111 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone6 from 0xe3c9d86 (overlap 0x2f8918) to 0x981d1 2024-09-16T13:28:40.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:40.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:28:40.157 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:40.155+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:40.226 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.227 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:40.270 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-09-16T13:28:40.270 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:40.268+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:40.308 INFO:teuthology.orchestra.run.smithi076.stdout:760 write 0x1635dce thru 0x16395ce (0x3801 bytes) 2024-09-16T13:28:40.326 INFO:teuthology.orchestra.run.smithi076.stdout:761 read 0x2da0093 thru 0x2da1604 (0x1572 bytes) 2024-09-16T13:28:40.326 INFO:teuthology.orchestra.run.smithi076.stdout:763 write 0xa56d835 thru 0xa56eb1d (0x12e9 bytes) 2024-09-16T13:28:40.335 INFO:teuthology.orchestra.run.smithi076.stdout:767 trunc from 0xe3c9d86 to 0xb0e9ffa 2024-09-16T13:28:40.349 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:40.349 INFO:teuthology.orchestra.run.smithi076.stdout:771 write 0xaab294b thru 0xaab8070 (0x5726 bytes) 2024-09-16T13:28:40.351 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:40.349+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:40.353 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.353 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.355 INFO:teuthology.orchestra.run.smithi076.stdout:773 read 0x884d3e5 thru 0x8850951 (0x356d bytes) 2024-09-16T13:28:40.362 INFO:teuthology.orchestra.run.smithi076.stdout:775 writesame 0xbdbbf86 thru 0xbdc7ed0 data_size 0x1(0xbf4b bytes) 2024-09-16T13:28:40.362 INFO:teuthology.orchestra.run.smithi076.stdout:776 read 0x7feaccb thru 0x7feacd1 (0x7 bytes) 2024-09-16T13:28:40.362 INFO:teuthology.orchestra.run.smithi076.stdout:778 read 0x4f68502 thru 0x4f6b535 (0x3034 bytes) 2024-09-16T13:28:40.362 INFO:teuthology.orchestra.run.smithi076.stdout:779 read 0x662b2a5 thru 0x66317d6 (0x6532 bytes) 2024-09-16T13:28:40.363 INFO:teuthology.orchestra.run.smithi076.stdout:781 read 0x87859e9 thru 0x87881b7 (0x27cf bytes) 2024-09-16T13:28:40.363 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:40.363 INFO:teuthology.orchestra.run.smithi076.stdout:783 read 0x4a9ac8f thru 0x4aa1b46 (0x6eb8 bytes) 2024-09-16T13:28:40.363 INFO:teuthology.orchestra.run.smithi076.stdout:784 write 0x5101615 thru 0x5103d19 (0x2705 bytes) 2024-09-16T13:28:40.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:40.378 INFO:teuthology.orchestra.run.smithi076.stdout:786 write 0xe99e336 thru 0xe9a33d1 (0x509c bytes) 2024-09-16T13:28:40.384 INFO:teuthology.orchestra.run.smithi076.stdout:787 write 0xa77d7f4 thru 0xa77f29a (0x1aa7 bytes) 2024-09-16T13:28:40.391 INFO:teuthology.orchestra.run.smithi076.stdout:788 flatten 2024-09-16T13:28:40.452 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:40.450+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:40.539 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.539 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:40.593 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:40.591+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:40.654 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:40.652+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:40.747 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.747 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.754 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:40.753+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:40.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:40.856 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:40.854+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:40.933 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:40.933 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:40.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:40.957 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:40.955+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:28:41.075 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:41.075 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:41.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:41.094 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:41.093+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:41.096 INFO:teuthology.orchestra.run.smithi076.stdout:790 writesame 0x1ce9f77 thru 0x1cebcda data_size 0x2c(0x1d64 bytes) 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:792 read 0xd4501ef thru 0xd45d376 (0xd188 bytes) 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:41.104 INFO:teuthology.orchestra.run.smithi076.stdout:796 writesame 0x746874b thru 0x7470758 data_size 0x25(0x800e bytes) 2024-09-16T13:28:41.113 INFO:teuthology.orchestra.run.smithi076.stdout:798 punch from 0x83e782a to 0x83eeef7, (0x76cd bytes) 2024-09-16T13:28:41.113 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:41.113 INFO:teuthology.orchestra.run.smithi076.stdout:801 write 0xa0ae4c4 thru 0xa0b2979 (0x44b6 bytes) 2024-09-16T13:28:41.119 INFO:teuthology.orchestra.run.smithi076.stdout:802 punch from 0x787c5fd to 0x787ec2c, (0x262f bytes) 2024-09-16T13:28:41.120 INFO:teuthology.orchestra.run.smithi076.stdout:803 punch from 0xa41806a to 0xa418472, (0x408 bytes) 2024-09-16T13:28:41.120 INFO:teuthology.orchestra.run.smithi076.stdout:804 write 0x6a87492 thru 0x6a8dbfc (0x676b bytes) 2024-09-16T13:28:41.122 INFO:teuthology.orchestra.run.smithi076.stdout:805 write 0x4092f15 thru 0x409908b (0x6177 bytes) 2024-09-16T13:28:41.129 INFO:teuthology.orchestra.run.smithi076.stdout:806 writesame 0xcacc42e thru 0xcad5eb0 data_size 0x2d(0x9a83 bytes) 2024-09-16T13:28:41.135 INFO:teuthology.orchestra.run.smithi076.stdout:808 write 0x8e6ef49 thru 0x8e77e49 (0x8f01 bytes) 2024-09-16T13:28:41.143 INFO:teuthology.orchestra.run.smithi076.stdout:809 read 0x1443c04 thru 0x14454f8 (0x18f5 bytes) 2024-09-16T13:28:41.143 INFO:teuthology.orchestra.run.smithi076.stdout:810 clone 8 order 20 su 131072 sc 5 2024-09-16T13:28:41.159 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:41.157+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:41.187 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:41.187 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:41.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:41.259 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:41.257+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:41.294 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:41.294 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:41.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:41.387 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:41.385+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:41.442 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:41.442 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:41.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:41.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:41.461 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:41.459+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:41.562 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:41.560+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:41.573 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:41.588 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:41.588 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:41.591 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:41.591 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:41.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:41.603 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:41.603 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:41.617 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:41.618 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:41.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:41.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:41.649 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:41.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:41.663 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:41.661+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:41.665 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:41.665 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:41.680 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:41.681 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.0976591009999996, 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:28:41.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:28:41.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:41.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:41.746 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.0071286959999997, 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:28:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3536 4.3 e38/32 4:c24635eb:::rbd_data.11a976a8123d.0000000000000000:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.997967998, 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018037721, 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.752 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739179+0000", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.316e-06 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742916+0000", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037375210000000002 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742931+0000", 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5413999999999998e-05 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742941+0000", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4290000000000007e-06 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742984+0000", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3022000000000002e-05 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742999+0000", 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5279000000000001e-05 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757171+0000", 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014172034999999999 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757215+0000", 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3705000000000003e-05 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.9799995260000003, 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.5642373740000002, 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:28:41.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4939010919999998, 2024-09-16T13:28:41.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:28:41.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:28:41.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6939 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 983040~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3545374050000003, 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017717554999999999, 2024-09-16T13:28:41.770 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6939 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382608+0000", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382610+0000", 2024-09-16T13:28:41.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4330000000000001e-06 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382616+0000", 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382620+0000", 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6459999999999999e-06 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400226+0000", 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017606218 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400267+0000", 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0914000000000002e-05 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400325+0000", 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8378999999999998e-05 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.774 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6947 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3545144660000004, 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017826794, 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6947 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382631+0000", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.03e-07 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382633+0000", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0990000000000001e-06 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382641+0000", 2024-09-16T13:28:41.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3889999999999999e-06 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382644+0000", 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7740000000000002e-06 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400347+0000", 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017702215 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400390+0000", 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3071999999999997e-05 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400458+0000", 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7842000000000005e-05 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.779 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6950 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3544967850000003, 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017927169999999999, 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6950 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382648+0000", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382649+0000", 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.41e-07 2024-09-16T13:28:41.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382656+0000", 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8220000000000001e-06 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382663+0000", 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1210000000000001e-06 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382666+0000", 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.534e-06 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400478+0000", 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017811421000000001 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400519+0000", 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0964999999999998e-05 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400576+0000", 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6966000000000003e-05 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6960 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3544750480000003, 2024-09-16T13:28:41.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018031049, 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6960 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:41.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382671+0000", 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382672+0000", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.601e-06 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382678+0000", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382681+0000", 2024-09-16T13:28:41.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400597+0000", 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017916527000000002 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400640+0000", 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2394000000000001e-05 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400701+0000", 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1334000000000006e-05 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6962 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3544603750000004, 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018139466, 2024-09-16T13:28:41.789 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6962 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382687+0000", 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382694+0000", 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7800000000000003e-06 2024-09-16T13:28:41.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382697+0000", 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400722+0000", 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018024563 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400763+0000", 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1003e-05 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400824+0000", 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1496999999999998e-05 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6968 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.794 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3544444420000001, 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018252758000000001, 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6968 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382703+0000", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382712+0000", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9600000000000006e-06 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382715+0000", 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5640000000000001e-06 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400844+0000", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018129125999999999 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400893+0000", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8869000000000003e-05 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400954+0000", 2024-09-16T13:28:41.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.02e-05 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6970 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3544251190000001, 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018372506, 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6970 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:41.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382722+0000", 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382728+0000", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382751+0000", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3697e-05 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400983+0000", 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018231872999999999 2024-09-16T13:28:41.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401028+0000", 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5148999999999998e-05 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401093+0000", 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4222999999999998e-05 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6984 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3542911069999999, 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018365686999999999, 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6984 2024-09-16T13:28:41.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382855+0000", 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.06e-07 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382858+0000", 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382868+0000", 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9450000000000005e-06 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382877+0000", 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3829999999999998e-06 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401119+0000", 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018241384999999999 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401160+0000", 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1393999999999997e-05 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401220+0000", 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9558999999999997e-05 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6986 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3542622800000004, 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018458407, 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6986 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:28:41.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382886+0000", 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.159e-06 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382892+0000", 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3690000000000003e-06 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382896+0000", 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401242+0000", 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018346475000000001 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401282+0000", 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401341+0000", 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8906999999999997e-05 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.813 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6990 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3542452310000002, 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018563608999999998, 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6990 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:28:41.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382902+0000", 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382909+0000", 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382913+0000", 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.552e-06 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401365+0000", 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018452196000000001 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401405+0000", 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0525999999999998e-05 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401464+0000", 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8226000000000001e-05 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6991 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2555904~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:41.818 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3542283800000003, 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018669403000000001, 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6991 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:41.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382919+0000", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.671e-06 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382926+0000", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7460000000000003e-06 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:28:41.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401486+0000", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018556492000000001 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401525+0000", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9338999999999997e-05 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401586+0000", 2024-09-16T13:28:41.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1124000000000006e-05 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7002 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3542121009999999, 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018778276, 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.823 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7002 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382942+0000", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4149999999999999e-06 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382949+0000", 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2840000000000001e-06 2024-09-16T13:28:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382953+0000", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401609+0000", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018656005999999999 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401650+0000", 2024-09-16T13:28:41.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1077e-05 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401711+0000", 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1245000000000001e-05 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7003 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.3541881340000002, 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018881326, 2024-09-16T13:28:41.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7003 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382959+0000", 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.61e-06 2024-09-16T13:28:41.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382977+0000", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7997999999999999e-05 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382985+0000", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929999999999998e-06 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401733+0000", 2024-09-16T13:28:41.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018748602 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401775+0000", 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1751000000000001e-05 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401838+0000", 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3188000000000006e-05 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7028 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.353248443, 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018078818999999999, 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7028 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:41.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383898+0000", 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0249999999999999e-06 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383902+0000", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.357e-06 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383912+0000", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0356999999999999e-05 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383918+0000", 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3290000000000001e-06 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401861+0000", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017943489999999999 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401916+0000", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4722999999999998e-05 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401976+0000", 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537999999999998e-05 2024-09-16T13:28:41.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:41.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:41.836 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:41.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:41.838 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:41.835+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:41.937 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:41.936+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:41.976 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:41.976 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:41.976 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:41.976 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:41.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:42.038 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:42.036+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:42.086 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:42.086 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:42.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:42.139 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:42.138+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:42.237 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5473333280000006, 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:28:42.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:28:42.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5377618500000008, 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7493 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10027008~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.690388767, 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036308945000000002, 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7493 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543964+0000", 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0200000000000001e-07 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543971+0000", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2599999999999999e-06 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543975+0000", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9509999999999999e-06 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543977+0000", 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.902e-06 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.579991+0000", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036013993000000001 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580036+0000", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5521e-05 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580273+0000", 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000236116 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7494 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10944512~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6903733860000001, 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036610563999999998, 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7494 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:42.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543980+0000", 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.85e-07 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543983+0000", 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543985+0000", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580303+0000", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036318691 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580359+0000", 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5072999999999999e-05 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580590+0000", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023103699999999999 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6903654379999997, 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.255 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6903531679999997, 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.259 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:42.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:28:42.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6903423230000003, 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6903329090000003, 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.268 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:28:42.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:28:42.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8153 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9441097960000002, 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036085006000000003, 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8153 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290243+0000", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.99e-07 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290244+0000", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290247+0000", 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7859999999999998e-06 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290249+0000", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.809e-06 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.325803+0000", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035554047999999998 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.325925+0000", 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012166499999999999 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326328+0000", 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040269200000000001 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8154 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9441014490000001, 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036367568000000003, 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.276 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8154 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290252+0000", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4799999999999997e-07 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290255+0000", 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9239999999999999e-06 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290257+0000", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7859999999999999e-06 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326341+0000", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036084789999999999 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326384+0000", 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509999999999998e-05 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326619+0000", 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023450899999999999 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440937759999999, 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:42.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:28:42.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:28:42.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.944085431, 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:28:42.284 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.288 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440747370000002, 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.292 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.944065749, 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:42.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:28:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:28:42.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.296 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440580079999998, 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:42.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:28:42.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:28:42.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.300 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.944049449, 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:28:42.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:28:42.304 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.944040572, 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.305 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:28:42.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.309 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440267069999999, 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:28:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440182400000001, 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:28:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9440094370000001, 2024-09-16T13:28:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:28:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:28:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.323 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:42.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:42.325 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:42.322+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:42.326 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-09-16T13:28:42.326 INFO:teuthology.orchestra.run.smithi076.stdout:812 write 0x48cb2c0 thru 0x48ccd51 (0x1a92 bytes) 2024-09-16T13:28:42.326 INFO:teuthology.orchestra.run.smithi076.stdout:814 read 0x1185375 thru 0x118f01a (0x9ca6 bytes) 2024-09-16T13:28:42.326 INFO:teuthology.orchestra.run.smithi076.stdout:816 trunc from 0xe9a33d2 to 0xabe3ef3 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:817 write 0xa865925 thru 0xa86ded0 (0x85ac bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:818 trunc from 0xabe3ef3 to 0xcd7d30e 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:819 punch from 0xc09f6fc to 0xc0aa395, (0xac99 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:820 write 0xd90f082 thru 0xd917386 (0x8305 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:821 punch from 0x67a61dc to 0x67b2082, (0xbea6 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:823 write 0x4befdfd thru 0x4bf5872 (0x5a76 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:827 write 0xe6fe81 thru 0xe71273 (0x13f3 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:829 write 0x42ee00a thru 0x42fd9be (0xf9b5 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:834 read 0x14d331f thru 0x14db7ae (0x8490 bytes) 2024-09-16T13:28:42.327 INFO:teuthology.orchestra.run.smithi076.stdout:835 write 0x87b1108 thru 0x87b43d5 (0x32ce bytes) 2024-09-16T13:28:42.328 INFO:teuthology.orchestra.run.smithi076.stdout:837 write 0x35e17d8 thru 0x35eaffa (0x9823 bytes) 2024-09-16T13:28:42.328 INFO:teuthology.orchestra.run.smithi076.stdout:838 punch from 0xc12f005 to 0xc139af6, (0xaaf1 bytes) 2024-09-16T13:28:42.328 INFO:teuthology.orchestra.run.smithi076.stdout:840 writesame 0x8b8c14c thru 0x8b9466c data_size 0x1(0x8521 bytes) 2024-09-16T13:28:42.384 INFO:teuthology.orchestra.run.smithi076.stdout:842 read 0x5f5ad27 thru 0x5f67ca5 (0xcf7f bytes) 2024-09-16T13:28:42.394 INFO:teuthology.orchestra.run.smithi076.stdout:843 trunc from 0xd917387 to 0x46d6603 2024-09-16T13:28:42.424 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:42.423+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:42.430 INFO:teuthology.orchestra.run.smithi076.stdout:845 write 0xe085a5c thru 0xe0924ef (0xca94 bytes) 2024-09-16T13:28:42.433 INFO:teuthology.orchestra.run.smithi076.stdout:846 clone 9 order 20 su 65536 sc 15 2024-09-16T13:28:42.467 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:42.468 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:42.468 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:42.468 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:42.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:42.562 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:42.560+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:42.614 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:42.614 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:42.615 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:42.615 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:42.615 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:42.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:42.664 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:42.662+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:42.727 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:42.725+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.469801386, 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:28:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:28:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:28:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.768451188999997, 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:28:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:28:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.768328793000002, 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:28:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:28:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:28:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.768216404999997, 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:28:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:42.786 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.768153591000001, 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:42.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.768006161000002, 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.792 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:28:42.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:28:42.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.797 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767990978, 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:28:42.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:28:42.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:42.803 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767969931000003, 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:28:42.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767882450999998, 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:42.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:28:42.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.815 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767859942000001, 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:28:42.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.821 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767813615000001, 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:28:42.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:28:42.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.826 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767646857999999, 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:28:42.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:42.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:28:42.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.767632093000003, 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:28:42.832 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.422309775999999, 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.838 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:42.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.843 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.422272098000001, 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.844 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:28:42.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:28:42.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:28:42.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.422249284999999, 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.850 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:28:42.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.422164072000001, 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.856 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:42.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:42.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:28:42.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:42.861 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.1155851470000009, 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:28:42.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:28:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.0896769129999999, 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.0209117580000004, 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:42.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:28:42.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:42.874 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:42.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:42.876 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:42.874+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:42.974 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:42.974 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:42.974 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:42.974 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:42.976 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:42.975+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:42.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:43.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:28:43.108 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone8 from 0xe0924f0 (overlap 0x46d6603) to 0x0 2024-09-16T13:28:43.109 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:43.107+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:43.149 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:43.149 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:43.150 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:43.150 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:43.150 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:43.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:43.178 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:43.176+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:43.243 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-09-16T13:28:43.278 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:43.277+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:43.331 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:43.331 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:43.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.16166356, 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.332 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:28:43.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:28:43.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:28:43.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.1561879949999998, 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:28:43.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:43.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:28:43.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3958611879999996, 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:28:43.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.3793230630000002, 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.351 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8144 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3538944~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0367596800000003, 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041437515000000001, 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8144 2024-09-16T13:28:43.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-07 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290571+0000", 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9120000000000001e-06 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290578+0000", 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9779999999999999e-06 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290582+0000", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.331618+0000", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041035335999999999 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.331688+0000", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0128000000000003e-05 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332007+0000", 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031891500000000002 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8145 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0367424940000003, 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041847961000000003, 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8145 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:43.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290587+0000", 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.03e-07 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290589+0000", 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.086e-06 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290596+0000", 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1509999999999998e-06 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290600+0000", 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.692e-06 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332033+0000", 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041433316999999997 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332120+0000", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6847000000000004e-05 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332434+0000", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031446499999999999 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8146 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0367255340000003, 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042182996, 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8146 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:43.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290604+0000", 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290605+0000", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290612+0000", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.686e-06 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290616+0000", 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:28:43.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332462+0000", 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041846176999999998 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332534+0000", 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2398999999999996e-05 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332786+0000", 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025215300000000002 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8147 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0367094620000001, 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042474660999999997, 2024-09-16T13:28:43.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8147 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290620+0000", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290621+0000", 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290627+0000", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9020000000000004e-06 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290632+0000", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6609999999999999e-06 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332812+0000", 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042180278000000002 2024-09-16T13:28:43.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332854+0000", 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1857999999999998e-05 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333094+0000", 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000239905 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366931810000004, 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:28:43.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:28:43.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366762029999999, 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:28:43.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:28:43.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366596130000003, 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:28:43.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366372679999998, 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:43.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.387 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366192879999998, 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:43.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0366034839999996, 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:28:43.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0365879229999999, 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:28:43.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0365702089999997, 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:28:43.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:43.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:28:43.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:28:43.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.0365436260000003, 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.194989713, 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:28:43.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.99755014200000003, 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:43.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:28:43.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.98705139500000005, 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.420 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:28:43.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.424 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:43.425 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:43.426 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:43.423+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:43.526 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:43.524+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:43.562 INFO:teuthology.orchestra.run.smithi076.stdout:847 punch from 0x58fe36a to 0x590dd81, (0xfa17 bytes) 2024-09-16T13:28:43.566 INFO:teuthology.orchestra.run.smithi076.stdout:848 write 0x438b755 thru 0x438f926 (0x41d2 bytes) 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23113 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.567589+0000", 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.00048769299999999998, 2024-09-16T13:28:43.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00050697199999999996, 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23113 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567589+0000", 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567589+0000", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567590+0000", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567594+0000", 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999999e-06 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567604+0000", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0063e-05 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567609+0000", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567647+0000", 2024-09-16T13:28:43.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8785999999999998e-05 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.567725+0000", 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7441000000000006e-05 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.568089+0000", 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036426500000000001 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.574 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:43.574 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 1 2024-09-16T13:28:43.574 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:43.574 INFO:teuthology.orchestra.run.smithi076.stdout:851 write 0x7e0d566 thru 0x7e1c480 (0xef1b bytes) 2024-09-16T13:28:43.578 INFO:teuthology.orchestra.run.smithi076.stdout:852 read 0x7bf217e thru 0x7c01206 (0xf089 bytes) 2024-09-16T13:28:43.580 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:43.580 INFO:teuthology.orchestra.run.smithi076.stdout:853 trunc from 0xe0924f0 to 0xca1ba21 2024-09-16T13:28:43.605 INFO:teuthology.orchestra.run.smithi076.stdout:854 read 0x6be7d39 thru 0x6bf09da (0x8ca2 bytes) 2024-09-16T13:28:43.605 INFO:teuthology.orchestra.run.smithi076.stdout:855 read 0x5132d9a thru 0x5136966 (0x3bcd bytes) 2024-09-16T13:28:43.605 INFO:teuthology.orchestra.run.smithi076.stdout:856 write 0x74fc667 thru 0x74fdfe5 (0x197f bytes) 2024-09-16T13:28:43.608 INFO:teuthology.orchestra.run.smithi076.stdout:857 read 0xb6c9a9e thru 0xb6d39d1 (0x9f34 bytes) 2024-09-16T13:28:43.608 INFO:teuthology.orchestra.run.smithi076.stdout:858 write 0xbfe6e60 thru 0xbfea6bc (0x385d bytes) 2024-09-16T13:28:43.611 INFO:teuthology.orchestra.run.smithi076.stdout:861 punch from 0xbb8cc59 to 0xbb91c9f, (0x5046 bytes) 2024-09-16T13:28:43.611 INFO:teuthology.orchestra.run.smithi076.stdout:862 write 0x4e7a4ea thru 0x4e80b10 (0x6627 bytes) 2024-09-16T13:28:43.617 INFO:teuthology.orchestra.run.smithi076.stdout:863 writesame 0xa4c8de6 thru 0xa4d5339 data_size 0x49(0xc554 bytes) 2024-09-16T13:28:43.627 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:43.625+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:43.628 INFO:teuthology.orchestra.run.smithi076.stdout:864 trunc from 0xca1ba21 to 0x61a3904 2024-09-16T13:28:43.667 INFO:teuthology.orchestra.run.smithi076.stdout:865 writesame 0xa9a6f41 thru 0xa9ae1b9 data_size 0x1(0x7279 bytes) 2024-09-16T13:28:43.667 INFO:teuthology.orchestra.run.smithi076.stdout:867 write 0xa14c8ba thru 0xa1577d3 (0xaf1a bytes) 2024-09-16T13:28:43.672 INFO:teuthology.orchestra.run.smithi076.stdout:868 punch from 0x9854e62 to 0x985b383, (0x6521 bytes) 2024-09-16T13:28:43.672 INFO:teuthology.orchestra.run.smithi076.stdout:870 read 0x6e1489f thru 0x6e1e299 (0x99fb bytes) 2024-09-16T13:28:43.672 INFO:teuthology.orchestra.run.smithi076.stdout:871 write 0x7975eea thru 0x797e10b (0x8222 bytes) 2024-09-16T13:28:43.675 INFO:teuthology.orchestra.run.smithi076.stdout:872 write 0x6cbef1d thru 0x6cc48f8 (0x59dc bytes) 2024-09-16T13:28:43.681 INFO:teuthology.orchestra.run.smithi076.stdout:874 read 0x5152379 thru 0x51581f5 (0x5e7d bytes) 2024-09-16T13:28:43.690 INFO:teuthology.orchestra.run.smithi076.stdout:875 write 0xd828fac thru 0xd836711 (0xd766 bytes) 2024-09-16T13:28:43.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:43.695 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:43.695 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:43.695 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:43.695 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:43.695 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:43.697 INFO:teuthology.orchestra.run.smithi076.stdout:876 writesame 0x607ca8d thru 0x608244c data_size 0x20(0x59c0 bytes) 2024-09-16T13:28:43.699 INFO:teuthology.orchestra.run.smithi076.stdout:879 read 0xd62ed66 thru 0xd6362b2 (0x754d bytes) 2024-09-16T13:28:43.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:43.706 INFO:teuthology.orchestra.run.smithi076.stdout:881 write 0xe5a043a thru 0xe5a3905 (0x34cc bytes) 2024-09-16T13:28:43.709 INFO:teuthology.orchestra.run.smithi076.stdout:882 read 0x9a812ed thru 0x9a89b0e (0x8822 bytes) 2024-09-16T13:28:43.709 INFO:teuthology.orchestra.run.smithi076.stdout:883 write 0x8e22474 thru 0x8e24486 (0x2013 bytes) 2024-09-16T13:28:43.712 INFO:teuthology.orchestra.run.smithi076.stdout:884 write 0xda0eca3 thru 0xda16887 (0x7be5 bytes) 2024-09-16T13:28:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:885 write 0x2da5900 thru 0x2da929d (0x399e bytes) 2024-09-16T13:28:43.719 INFO:teuthology.orchestra.run.smithi076.stdout:886 write 0xc7c58b8 thru 0xc7d27ba (0xcf03 bytes) 2024-09-16T13:28:43.725 INFO:teuthology.orchestra.run.smithi076.stdout:888 writesame 0x6ac159d thru 0x6ac43b6 data_size 0x15(0x2e1a bytes) 2024-09-16T13:28:43.728 INFO:teuthology.orchestra.run.smithi076.stdout:890 write 0x2c2a637 thru 0x2c31fa2 (0x796c bytes) 2024-09-16T13:28:43.728 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:43.725+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:893 read 0x19c5597 thru 0x19cf8db (0xa345 bytes) 2024-09-16T13:28:43.747 INFO:teuthology.orchestra.run.smithi076.stdout:894 trunc from 0xe5a3906 to 0x38ec753 2024-09-16T13:28:43.796 INFO:teuthology.orchestra.run.smithi076.stdout:895 punch from 0x30fe707 to 0x310b367, (0xcc60 bytes) 2024-09-16T13:28:43.796 INFO:teuthology.orchestra.run.smithi076.stdout:896 read 0x146236 thru 0x14bdf9 (0x5bc4 bytes) 2024-09-16T13:28:43.798 INFO:teuthology.orchestra.run.smithi076.stdout:898 trunc from 0x38ec753 to 0x1c20a0e 2024-09-16T13:28:43.829 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:43.827+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.390736025000001, 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:28:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:1146 4.f e37/32 ondisk, result = 0)", 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.387769925000001, 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014045078000000001, 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445311+0000", 2024-09-16T13:28:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6499999999999998e-07 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445315+0000", 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0319999999999997e-06 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445327+0000", 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2495e-05 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445333+0000", 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9449999999999998e-06 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459267+0000", 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013933555 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459277+0000", 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0594000000000001e-05 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459355+0000", 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7591999999999998e-05 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.2183368489999999, 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:28:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:28:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3373 4.b e38/32 4:d151a9d0:::rbd_data.11a976a8123d.000000000000000a:head v 38'14, mlcod=38'11)", 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1270335459999998, 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014151797000000001, 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706047+0000", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706592+0000", 2024-09-16T13:28:43.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054438299999999995 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706613+0000", 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1118999999999999e-05 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706620+0000", 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706670+0000", 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280999999999998e-05 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706697+0000", 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7452000000000001e-05 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720145+0000", 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013447506999999999 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720198+0000", 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3232999999999997e-05 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7580 4.2 4:467b4c68:::rbd_data.11a91a08d9ca.0000000000000019:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.656496+0000", 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.176583634, 2024-09-16T13:28:43.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012346762000000001, 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7580 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656496+0000", 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656496+0000", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656497+0000", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7400000000000003e-07 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656500+0000", 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4989999999999999e-06 2024-09-16T13:28:43.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656509+0000", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7600000000000008e-06 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656513+0000", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.63e-06 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656546+0000", 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3757000000000002e-05 2024-09-16T13:28:43.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.656595+0000", 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8461999999999998e-05 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.657224+0000", 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062919899999999999 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.668773+0000", 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01154876 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.668780+0000", 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7430000000000002e-06 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.668817+0000", 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6575000000000001e-05 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.668843+0000", 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5703000000000001e-05 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.862 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.983264975, 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7662 4.12 e41/32 ondisk, result = 0)", 2024-09-16T13:28:43.868 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.855692+0000", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.9773880620000002, 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012919784, 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855692+0000", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855692+0000", 2024-09-16T13:28:43.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855692+0000", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855696+0000", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6229999999999999e-06 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855706+0000", 2024-09-16T13:28:43.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0328e-05 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.855710+0000", 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.352e-06 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868514+0000", 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012803419999999999 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868521+0000", 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.277e-06 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868611+0000", 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0143999999999996e-05 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.872 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.1305648530000001, 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:43.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:28:43.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:43.878 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.1095009449999997, 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:28:43.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.8054279070000003, 2024-09-16T13:28:43.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:28:43.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.700834085, 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.888 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:43.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:28:43.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21711 4.4 4:2249ef24:::rbd_object_map.11a98415fd69.000000000000000c:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.136116+0000", 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.69696381399999996, 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01273113, 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21711 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.893 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136116+0000", 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136116+0000", 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136117+0000", 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136124+0000", 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6449999999999999e-06 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136136+0000", 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2319e-05 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136151+0000", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5145e-05 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136230+0000", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8263999999999995e-05 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136318+0000", 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8652000000000004e-05 2024-09-16T13:28:43.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136817+0000", 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049841800000000004 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141807+0000", 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049905089999999997 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.148759+0000", 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069515779999999999 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.148810+0000", 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0992999999999998e-05 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.148847+0000", 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7159000000000001e-05 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21707 4.14 e46/32 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head v 46'74, mlcod=45'72)", 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.69609085199999998, 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015037457000000001, 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:43.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000003e-07 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137531+0000", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054107700000000005 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137543+0000", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2156e-05 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137548+0000", 2024-09-16T13:28:43.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.463e-06 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137896+0000", 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034767700000000001 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137917+0000", 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1523999999999999e-05 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.151679+0000", 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013761997 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152026+0000", 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034681000000000002 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:43.903 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.69429044399999995, 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:43.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:28:43.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:28:43.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23128 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.580286+0000", 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.25279317499999998, 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012677745000000001, 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.908 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23128 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580286+0000", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580286+0000", 2024-09-16T13:28:43.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580287+0000", 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7399999999999998e-07 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580290+0000", 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.281e-06 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580306+0000", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5361000000000001e-05 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580310+0000", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4129999999999999e-06 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.586866+0000", 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065560699999999998 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.586942+0000", 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6328000000000005e-05 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.587288+0000", 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034522399999999999 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589783+0000", 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002495007 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.592909+0000", 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031263520000000002 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.592934+0000", 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5273999999999999e-05 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.592964+0000", 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9961e-05 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.914 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23129 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.580322+0000", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.25275772200000002, 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012737584, 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23129 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580322+0000", 2024-09-16T13:28:43.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580322+0000", 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580322+0000", 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7e-07 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580326+0000", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580336+0000", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0395e-05 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580340+0000", 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0910000000000003e-06 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.587612+0000", 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007272026 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.587704+0000", 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2108000000000006e-05 2024-09-16T13:28:43.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.588078+0000", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037344800000000002 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589789+0000", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017109989999999999 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.592994+0000", 2024-09-16T13:28:43.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032049679999999999 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593017+0000", 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3334000000000001e-05 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593059+0000", 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2449e-05 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23130 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.580363+0000", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.25271661200000001, 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012799400000000001, 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23130 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580363+0000", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580363+0000", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580364+0000", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3600000000000004e-07 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580367+0000", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3179999999999998e-06 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580373+0000", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2700000000000001e-06 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580377+0000", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1309999999999999e-06 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.588417+0000", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008040017 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.588466+0000", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8789999999999999e-05 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.588699+0000", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000233377 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.590701+0000", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002001451 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593100+0000", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023988569999999999 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593124+0000", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4729999999999999e-05 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593162+0000", 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7923000000000003e-05 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23131 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.580412+0000", 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.25266747299999998, 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012854399000000001, 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23131 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580412+0000", 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580412+0000", 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580413+0000", 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4400000000000001e-07 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580421+0000", 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9220000000000003e-06 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580430+0000", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8580000000000006e-06 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580435+0000", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3159999999999998e-06 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589163+0000", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087280829999999993 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589247+0000", 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4101e-05 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589498+0000", 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025082899999999999 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.590850+0000", 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013524959999999999 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593199+0000", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023491279999999998 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593239+0000", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9746999999999997e-05 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593267+0000", 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7475e-05 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23132 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.580441+0000", 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.25263812699999999, 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012947699, 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23132 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580441+0000", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580441+0000", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580442+0000", 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:28:43.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580445+0000", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.943e-06 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580451+0000", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1589999999999998e-06 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.580455+0000", 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8510000000000001e-06 2024-09-16T13:28:43.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589828+0000", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009373167 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.589906+0000", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7590999999999996e-05 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.590275+0000", 2024-09-16T13:28:43.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036915299999999999 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.592877+0000", 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026025250000000001 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593303+0000", 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042615799999999999 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593342+0000", 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8387999999999999e-05 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.593389+0000", 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7314999999999997e-05 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23815 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.814815+0000", 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.018264740000000002, 2024-09-16T13:28:43.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012420511, 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23815 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814815+0000", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:43.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814815+0000", 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814816+0000", 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.08e-06 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814821+0000", 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9899999999999997e-06 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814833+0000", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2140999999999999e-05 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.814842+0000", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4780000000000008e-06 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.821862+0000", 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070201530000000003 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.821935+0000", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3462999999999998e-05 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.822130+0000", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019510099999999999 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824293+0000", 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021627069999999998 2024-09-16T13:28:43.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827174+0000", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028813100000000002 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827201+0000", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6990000000000001e-05 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827235+0000", 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4097999999999998e-05 2024-09-16T13:28:43.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:43.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:43.943 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:43.944 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:43.944 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:43.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:43.945 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:43.946 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:43.946 INFO:teuthology.orchestra.run.smithi076.stdout:899 write 0xbce121e thru 0xbce28a5 (0x1688 bytes) 2024-09-16T13:28:43.946 INFO:teuthology.orchestra.run.smithi076.stdout:901 trunc from 0xbce28a6 to 0x4c2ee86 2024-09-16T13:28:43.946 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:43.946 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:43.947 INFO:teuthology.orchestra.run.smithi076.stdout:902 read 0x2e92f64 thru 0x2e965dd (0x367a bytes) 2024-09-16T13:28:43.947 INFO:teuthology.orchestra.run.smithi076.stdout:903 write 0x3b1bb9 thru 0x3bfbf5 (0xe03d bytes) 2024-09-16T13:28:43.947 INFO:teuthology.orchestra.run.smithi076.stdout:904 read 0x21e576d thru 0x21e7d84 (0x2618 bytes) 2024-09-16T13:28:43.947 INFO:teuthology.orchestra.run.smithi076.stdout:905 punch from 0x3658ef6 to 0x365b056, (0x2160 bytes) 2024-09-16T13:28:43.947 INFO:teuthology.orchestra.run.smithi076.stdout:906 write 0x8d4beb thru 0x8d7605 (0x2a1b bytes) 2024-09-16T13:28:43.948 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:43.948 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:43.948 INFO:teuthology.orchestra.run.smithi076.stdout:907 read 0x372de12 thru 0x3730d9b (0x2f8a bytes) 2024-09-16T13:28:43.948 INFO:teuthology.orchestra.run.smithi076.stdout:909 read 0x1597d49 thru 0x15a5057 (0xd30f bytes) 2024-09-16T13:28:43.948 INFO:teuthology.orchestra.run.smithi076.stdout:910 writesame 0x764afc1 thru 0x76584e7 data_size 0x2f(0xd527 bytes) 2024-09-16T13:28:43.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:43.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:43.949 INFO:teuthology.orchestra.run.smithi076.stdout:911 writesame 0xb49dd59 thru 0xb4aad35 data_size 0x1(0xcfdd bytes) 2024-09-16T13:28:43.949 INFO:teuthology.orchestra.run.smithi076.stdout:912 writesame 0xace38ec thru 0xace93be data_size 0x1(0x5ad3 bytes) 2024-09-16T13:28:43.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:43.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:43.950 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:43.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:43.950 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:43.941+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:43.956 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:43.956 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:43.960 INFO:teuthology.orchestra.run.smithi076.stdout:913 punch from 0x838e2ff to 0x8391f1b, (0x3c1c bytes) 2024-09-16T13:28:43.960 INFO:teuthology.orchestra.run.smithi076.stdout:914 read 0xab02390 thru 0xab07af0 (0x5761 bytes) 2024-09-16T13:28:43.960 INFO:teuthology.orchestra.run.smithi076.stdout:915 read 0x61e3474 thru 0x61f3055 (0xfbe2 bytes) 2024-09-16T13:28:43.961 INFO:teuthology.orchestra.run.smithi076.stdout:916 writesame 0x4cbde8f thru 0x4ccb00f data_size 0x1(0xd181 bytes) 2024-09-16T13:28:44.029 INFO:teuthology.orchestra.run.smithi076.stdout:917 punch from 0x47891de to 0x47944ae, (0xb2d0 bytes) 2024-09-16T13:28:44.036 INFO:teuthology.orchestra.run.smithi076.stdout:919 write 0xc4ee1c5 thru 0xc4f4267 (0x60a3 bytes) 2024-09-16T13:28:44.043 INFO:teuthology.orchestra.run.smithi076.stdout:920 write 0xa446439 thru 0xa44f49f (0x9067 bytes) 2024-09-16T13:28:44.045 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:44.043+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:44.048 INFO:teuthology.orchestra.run.smithi076.stdout:921 trunc from 0xc4f4268 to 0xc45e203 2024-09-16T13:28:44.055 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:28:44.058 INFO:teuthology.orchestra.run.smithi076.stdout:922 punch from 0x4080c19 to 0x408f998, (0xed7f bytes) 2024-09-16T13:28:44.062 INFO:teuthology.orchestra.run.smithi076.stdout:923 trunc from 0xc45e203 to 0x8e0a421 2024-09-16T13:28:44.065 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.066 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.072 INFO:teuthology.orchestra.run.smithi076.stdout:924 write 0x15261c6 thru 0x1535686 (0xf4c1 bytes) 2024-09-16T13:28:44.075 INFO:teuthology.orchestra.run.smithi076.stdout:926 read 0x35dd3c9 thru 0x35e3c9c (0x68d4 bytes) 2024-09-16T13:28:44.075 INFO:teuthology.orchestra.run.smithi076.stdout:929 write 0x7538527 thru 0x753c6d7 (0x41b1 bytes) 2024-09-16T13:28:44.076 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:44.078 INFO:teuthology.orchestra.run.smithi076.stdout:931 write 0x6d7ced8 thru 0x6d85eb6 (0x8fdf bytes) 2024-09-16T13:28:44.093 INFO:teuthology.orchestra.run.smithi076.stdout:932 write 0xe1eca2b thru 0xe1fc7a8 (0xfd7e bytes) 2024-09-16T13:28:44.097 INFO:teuthology.orchestra.run.smithi076.stdout:933 read 0x42bd6d9 thru 0x42bf149 (0x1a71 bytes) 2024-09-16T13:28:44.097 INFO:teuthology.orchestra.run.smithi076.stdout:934 writesame 0x6d1f517 thru 0x6d24843 data_size 0x1(0x532d bytes) 2024-09-16T13:28:44.112 INFO:teuthology.orchestra.run.smithi076.stdout:937 read 0xa84daec thru 0xa851304 (0x3819 bytes) 2024-09-16T13:28:44.114 INFO:teuthology.orchestra.run.smithi076.stdout:938 trunc from 0xe1fc7a9 to 0xc938275 2024-09-16T13:28:44.123 INFO:teuthology.orchestra.run.smithi076.stdout:940 write 0xc8c78a4 thru 0xc8cfb24 (0x8281 bytes) 2024-09-16T13:28:44.127 INFO:teuthology.orchestra.run.smithi076.stdout:941 writesame 0x253cece thru 0x25436ec data_size 0x5(0x681f bytes) 2024-09-16T13:28:44.132 INFO:teuthology.orchestra.run.smithi076.stdout:942 punch from 0x111d3 to 0x1e1b7, (0xcfe4 bytes) 2024-09-16T13:28:44.132 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.132 INFO:teuthology.orchestra.run.smithi076.stdout:945 read 0xba2c1dc thru 0xba35ed0 (0x9cf5 bytes) 2024-09-16T13:28:44.132 INFO:teuthology.orchestra.run.smithi076.stdout:946 read 0x1ecbdef thru 0x1ed8123 (0xc335 bytes) 2024-09-16T13:28:44.132 INFO:teuthology.orchestra.run.smithi076.stdout:947 read 0xc0c1868 thru 0xc0d0000 (0xe799 bytes) 2024-09-16T13:28:44.137 INFO:teuthology.orchestra.run.smithi076.stdout:950 writesame 0xd3b2dfc thru 0xd3b54be data_size 0x1(0x26c3 bytes) 2024-09-16T13:28:44.138 INFO:teuthology.orchestra.run.smithi076.stdout:951 writesame 0x89e00f7 thru 0x89ebefd data_size 0x1(0xbe07 bytes) 2024-09-16T13:28:44.146 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:44.144+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:44.188 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.189 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.198 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:44.219 INFO:teuthology.orchestra.run.smithi076.stdout:953 writesame 0x1f0b66d thru 0x1f15b68 data_size 0x2(0xa4fc bytes) 2024-09-16T13:28:44.230 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.230 INFO:teuthology.orchestra.run.smithi076.stdout:955 read 0xb651e03 thru 0xb65d583 (0xb781 bytes) 2024-09-16T13:28:44.230 INFO:teuthology.orchestra.run.smithi076.stdout:957 read 0xcec4b32 thru 0xcec97f2 (0x4cc1 bytes) 2024-09-16T13:28:44.230 INFO:teuthology.orchestra.run.smithi076.stdout:959 write 0xbc38f6a thru 0xbc43d36 (0xadcd bytes) 2024-09-16T13:28:44.247 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:44.245+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:44.249 INFO:teuthology.orchestra.run.smithi076.stdout:961 trunc from 0xd3b54bf to 0xe2942a 2024-09-16T13:28:44.296 INFO:teuthology.orchestra.run.smithi076.stdout:962 trunc from 0xe2942a to 0x63deec8 2024-09-16T13:28:44.301 INFO:teuthology.orchestra.run.smithi076.stdout:963 write 0x101c6e6 thru 0x1029acd (0xd3e8 bytes) 2024-09-16T13:28:44.305 INFO:teuthology.orchestra.run.smithi076.stdout:964 trunc from 0x63deec8 to 0x6335842 2024-09-16T13:28:44.310 INFO:teuthology.orchestra.run.smithi076.stdout:965 writesame 0x61da3e7 thru 0x61e8093 data_size 0x3(0xdcad bytes) 2024-09-16T13:28:44.337 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.337 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.348 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:44.346+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:44.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:44.351 INFO:teuthology.orchestra.run.smithi076.stdout:967 writesame 0xd1fff67 thru 0xd203820 data_size 0x2(0x38ba bytes) 2024-09-16T13:28:44.356 INFO:teuthology.orchestra.run.smithi076.stdout:968 write 0xdbeba04 thru 0xdbf10b3 (0x56b0 bytes) 2024-09-16T13:28:44.363 INFO:teuthology.orchestra.run.smithi076.stdout:969 writesame 0xbb4570e thru 0xbb54f5f data_size 0x1e(0xf852 bytes) 2024-09-16T13:28:44.369 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.369 INFO:teuthology.orchestra.run.smithi076.stdout:976 read 0x4878186 thru 0x488135c (0x91d7 bytes) 2024-09-16T13:28:44.369 INFO:teuthology.orchestra.run.smithi076.stdout:977 punch from 0xc988947 to 0xc995135, (0xc7ee bytes) 2024-09-16T13:28:44.369 INFO:teuthology.orchestra.run.smithi076.stdout:978 writesame 0x8aeb9a9 thru 0x8aef99b data_size 0x11(0x3ff3 bytes) 2024-09-16T13:28:44.373 INFO:teuthology.orchestra.run.smithi076.stdout:979 read 0x1b7c01 thru 0x1c5f0b (0xe30b bytes) 2024-09-16T13:28:44.374 INFO:teuthology.orchestra.run.smithi076.stdout:981 write 0x7d852fa thru 0x7d8cc16 (0x791d bytes) 2024-09-16T13:28:44.390 INFO:teuthology.orchestra.run.smithi076.stdout:982 trunc from 0xdbf10b4 to 0x4acf6fe 2024-09-16T13:28:44.402 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.410 INFO:teuthology.orchestra.run.smithi076.stdout:984 write 0x676cad7 thru 0x67783be (0xb8e8 bytes) 2024-09-16T13:28:44.415 INFO:teuthology.orchestra.run.smithi076.stdout:985 writesame 0x2bce308 thru 0x2bcff39 data_size 0x3(0x1c32 bytes) 2024-09-16T13:28:44.419 INFO:teuthology.orchestra.run.smithi076.stdout:986 read 0x5752635 thru 0x57567d3 (0x419f bytes) 2024-09-16T13:28:44.419 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.419 INFO:teuthology.orchestra.run.smithi076.stdout:989 punch from 0x38a0c94 to 0x38b016b, (0xf4d7 bytes) 2024-09-16T13:28:44.426 INFO:teuthology.orchestra.run.smithi076.stdout:991 trunc from 0x67783bf to 0x5293c4 2024-09-16T13:28:44.450 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:44.447+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:44.466 INFO:teuthology.orchestra.run.smithi076.stdout:992 writesame 0xd5f2a51 thru 0xd5f40cc data_size 0x2(0x167c bytes) 2024-09-16T13:28:44.466 INFO:teuthology.orchestra.run.smithi076.stdout:994 read 0xbed3ab1 thru 0xbed5a72 (0x1fc2 bytes) 2024-09-16T13:28:44.466 INFO:teuthology.orchestra.run.smithi076.stdout:995 read 0x8ac5a36 thru 0x8ac6f93 (0x155e bytes) 2024-09-16T13:28:44.466 INFO:teuthology.orchestra.run.smithi076.stdout:996 write 0x78b1ef5 thru 0x78b4e7a (0x2f86 bytes) 2024-09-16T13:28:44.469 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.469 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.470 INFO:teuthology.orchestra.run.smithi076.stdout:998 punch from 0x78d4cf5 to 0x78ddf6f, (0x927a bytes) 2024-09-16T13:28:44.470 INFO:teuthology.orchestra.run.smithi076.stdout:1001 write 0x91396ef thru 0x9147bb0 (0xe4c2 bytes) 2024-09-16T13:28:44.475 INFO:teuthology.orchestra.run.smithi076.stdout:1002 writesame 0xa4d6647 thru 0xa4e0a6e data_size 0x44(0xa428 bytes) 2024-09-16T13:28:44.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:44.483 INFO:teuthology.orchestra.run.smithi076.stdout:1003 write 0x5c2cbe5 thru 0x5c3a4fb (0xd917 bytes) 2024-09-16T13:28:44.490 INFO:teuthology.orchestra.run.smithi076.stdout:1006 write 0xaccb695 thru 0xaccc99b (0x1307 bytes) 2024-09-16T13:28:44.493 INFO:teuthology.orchestra.run.smithi076.stdout:1007 writesame 0x7732334 thru 0x77324f8 data_size 0x3(0x1c5 bytes) 2024-09-16T13:28:44.500 INFO:teuthology.orchestra.run.smithi076.stdout:1008 writesame 0xd6b94ce thru 0xd6c42a9 data_size 0x6(0xaddc bytes) 2024-09-16T13:28:44.500 INFO:teuthology.orchestra.run.smithi076.stdout:1009 write 0xcad36c3 thru 0xcad58bc (0x21fa bytes) 2024-09-16T13:28:44.502 INFO:teuthology.orchestra.run.smithi076.stdout:1010 read 0x239537e thru 0x23979cc (0x264f bytes) 2024-09-16T13:28:44.514 INFO:teuthology.orchestra.run.smithi076.stdout:1011 trunc from 0xd6c42aa to 0x360ea64 2024-09-16T13:28:44.524 INFO:teuthology.orchestra.run.smithi076.stdout:1012 punch from 0x313f812 to 0x3145032, (0x5820 bytes) 2024-09-16T13:28:44.524 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.538 INFO:teuthology.orchestra.run.smithi076.stdout:1015 write 0xa5a6718 thru 0xa5b59a2 (0xf28b bytes) 2024-09-16T13:28:44.542 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.542 INFO:teuthology.orchestra.run.smithi076.stdout:1017 read 0x57693e3 thru 0x576a506 (0x1124 bytes) 2024-09-16T13:28:44.542 INFO:teuthology.orchestra.run.smithi076.stdout:1018 punch from 0x1ef568c to 0x1efc9e2, (0x7356 bytes) 2024-09-16T13:28:44.542 INFO:teuthology.orchestra.run.smithi076.stdout:1020 write 0x420fe4e thru 0x421fd51 (0xff04 bytes) 2024-09-16T13:28:44.549 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.549 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.549 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.550 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.550 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.550 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:44.548+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:44.553 INFO:teuthology.orchestra.run.smithi076.stdout:1033 trunc from 0xa5b59a3 to 0x76fbae5 2024-09-16T13:28:44.561 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.562 INFO:teuthology.orchestra.run.smithi076.stdout:1038 trunc from 0x76fbae5 to 0x7022ec0 2024-09-16T13:28:44.567 INFO:teuthology.orchestra.run.smithi076.stdout:1039 read 0x51a507 thru 0x5231f1 (0x8ceb bytes) 2024-09-16T13:28:44.574 INFO:teuthology.orchestra.run.smithi076.stdout:1040 write 0x7c51c94 thru 0x7c5f6b1 (0xda1e bytes) 2024-09-16T13:28:44.577 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.577 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.577 INFO:teuthology.orchestra.run.smithi076.stdout:1044 punch from 0x68cdf5b to 0x68d753b, (0x95e0 bytes) 2024-09-16T13:28:44.582 INFO:teuthology.orchestra.run.smithi076.stdout:1046 writesame 0x8411f36 thru 0x841dfdc data_size 0x1(0xc0a7 bytes) 2024-09-16T13:28:44.582 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xeb793c2 2024-09-16T13:28:44.590 INFO:teuthology.orchestra.run.smithi076.stdout:1047 trunc from 0x841dfdd to 0xeb793c2 2024-09-16T13:28:44.595 INFO:teuthology.orchestra.run.smithi076.stdout:1048 writesame 0x6553811 thru 0x6562d7e data_size 0x7a(0xf56e bytes) 2024-09-16T13:28:44.598 INFO:teuthology.orchestra.run.smithi076.stdout:1052 writesame 0xb6d3646 thru 0xb6e1d4f data_size 0x1(0xe70a bytes) 2024-09-16T13:28:44.604 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.604 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:44.651 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:44.649+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:44.704 INFO:teuthology.orchestra.run.smithi076.stdout:1055 writesame 0x326b201 thru 0x326f77d data_size 0x1(0x457d bytes) 2024-09-16T13:28:44.719 INFO:teuthology.orchestra.run.smithi076.stdout:1057 write 0xa8c58d7 thru 0xa8c6685 (0xdaf bytes) 2024-09-16T13:28:44.722 INFO:teuthology.orchestra.run.smithi076.stdout:1058 write 0x4b806c4 thru 0x4b904f5 (0xfe32 bytes) 2024-09-16T13:28:44.728 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.728 INFO:teuthology.orchestra.run.smithi076.stdout:1061 read 0x955a258 thru 0x956405d (0x9e06 bytes) 2024-09-16T13:28:44.728 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.728 INFO:teuthology.orchestra.run.smithi076.stdout:1063 write 0x9e86a07 thru 0x9e94dc5 (0xe3bf bytes) 2024-09-16T13:28:44.733 INFO:teuthology.orchestra.run.smithi076.stdout:1064 writesame 0x4b97473 thru 0x4ba6148 data_size 0x2f(0xecd6 bytes) 2024-09-16T13:28:44.737 INFO:teuthology.orchestra.run.smithi076.stdout:1065 punch from 0x61d85 to 0x67111, (0x538c bytes) 2024-09-16T13:28:44.737 INFO:teuthology.orchestra.run.smithi076.stdout:1066 write 0x146dcd2 thru 0x147aa27 (0xcd56 bytes) 2024-09-16T13:28:44.749 INFO:teuthology.orchestra.run.smithi076.stdout:1067 trunc from 0xeb793c2 to 0x8779d8d 2024-09-16T13:28:44.751 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:44.748+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:44.760 INFO:teuthology.orchestra.run.smithi076.stdout:1070 read 0x10569a3 thru 0x105ef69 (0x85c7 bytes) 2024-09-16T13:28:44.760 INFO:teuthology.orchestra.run.smithi076.stdout:1073 write 0x73ff9ec thru 0x7401330 (0x1945 bytes) 2024-09-16T13:28:44.762 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.763 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:44.775 INFO:teuthology.orchestra.run.smithi076.stdout:1075 writesame 0xb086fd4 thru 0xb08913d data_size 0xe(0x216a bytes) 2024-09-16T13:28:44.775 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xecc6208 2024-09-16T13:28:44.778 INFO:teuthology.orchestra.run.smithi076.stdout:1076 trunc from 0xb08913e to 0xecc6208 2024-09-16T13:28:44.783 INFO:teuthology.orchestra.run.smithi076.stdout:1077 read 0x25ac178 thru 0x25b77fe (0xb687 bytes) 2024-09-16T13:28:44.783 INFO:teuthology.orchestra.run.smithi076.stdout:1078 writesame 0x238c8e1 thru 0x2399982 data_size 0x46(0xd0a2 bytes) 2024-09-16T13:28:44.787 INFO:teuthology.orchestra.run.smithi076.stdout:1079 read 0x7f4df3a thru 0x7f4fe39 (0x1f00 bytes) 2024-09-16T13:28:44.795 INFO:teuthology.orchestra.run.smithi076.stdout:1080 trunc from 0xecc6208 to 0x80076c1 2024-09-16T13:28:44.809 INFO:teuthology.orchestra.run.smithi076.stdout:1081 trunc from 0x80076c1 to 0x1461a55 2024-09-16T13:28:44.821 INFO:teuthology.orchestra.run.smithi076.stdout:1082 read 0x4810d0 thru 0x483431 (0x2362 bytes) 2024-09-16T13:28:44.822 INFO:teuthology.orchestra.run.smithi076.stdout:1083 read 0x8c73a4 thru 0x8d3193 (0xbdf0 bytes) 2024-09-16T13:28:44.822 INFO:teuthology.orchestra.run.smithi076.stdout:1085 read 0x109acf0 thru 0x10a334d (0x865e bytes) 2024-09-16T13:28:44.823 INFO:teuthology.orchestra.run.smithi076.stdout:1086 read 0xf43fc8 thru 0xf5268d (0xe6c6 bytes) 2024-09-16T13:28:44.843 INFO:teuthology.orchestra.run.smithi076.stdout:1087 write 0xd541686 thru 0xd542b45 (0x14c0 bytes) 2024-09-16T13:28:44.846 INFO:teuthology.orchestra.run.smithi076.stdout:1088 read 0x851c417 thru 0x85212ef (0x4ed9 bytes) 2024-09-16T13:28:44.846 INFO:teuthology.orchestra.run.smithi076.stdout:1089 read 0x7b3341 thru 0x7b40a3 (0xd63 bytes) 2024-09-16T13:28:44.847 INFO:teuthology.orchestra.run.smithi076.stdout:1092 punch from 0x4e9cd20 to 0x4ea46c6, (0x79a6 bytes) 2024-09-16T13:28:44.851 INFO:teuthology.orchestra.run.smithi076.stdout:1093 trunc from 0xd542b46 to 0x951b5cc 2024-09-16T13:28:44.852 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:44.849+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:44.874 INFO:teuthology.orchestra.run.smithi076.stdout:1094 write 0xeb4b829 thru 0xeb4fa4a (0x4222 bytes) 2024-09-16T13:28:44.877 INFO:teuthology.orchestra.run.smithi076.stdout:1095 write 0x220e333 thru 0x2211b70 (0x383e bytes) 2024-09-16T13:28:44.878 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.878 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.882 INFO:teuthology.orchestra.run.smithi076.stdout:1097 read 0x43beee8 thru 0x43c18dd (0x29f6 bytes) 2024-09-16T13:28:44.882 INFO:teuthology.orchestra.run.smithi076.stdout:1099 punch from 0x8019c8f to 0x8022045, (0x83b6 bytes) 2024-09-16T13:28:44.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:44.894 INFO:teuthology.orchestra.run.smithi076.stdout:1100 trunc from 0xeb4fa4b to 0x41aa834 2024-09-16T13:28:44.904 INFO:teuthology.orchestra.run.smithi076.stdout:1101 write 0x1c50de thru 0x1ce363 (0x9286 bytes) 2024-09-16T13:28:44.907 INFO:teuthology.orchestra.run.smithi076.stdout:1103 trunc from 0x41aa834 to 0x42f01b0 2024-09-16T13:28:44.920 INFO:teuthology.orchestra.run.smithi076.stdout:1104 writesame 0x68f0796 thru 0x68f66ff data_size 0x45(0x5f6a bytes) 2024-09-16T13:28:44.931 INFO:teuthology.orchestra.run.smithi076.stdout:1105 writesame 0xb139faa thru 0xb149c44 data_size 0x1(0xfc9b bytes) 2024-09-16T13:28:44.931 INFO:teuthology.orchestra.run.smithi076.stdout:1106 punch from 0xe39f44 to 0xe3a091, (0x14d bytes) 2024-09-16T13:28:44.931 INFO:teuthology.orchestra.run.smithi076.stdout:1107 punch from 0x4453f24 to 0x4461b47, (0xdc23 bytes) 2024-09-16T13:28:44.931 INFO:teuthology.orchestra.run.smithi076.stdout:1108 read 0x71b7c2 thru 0x724076 (0x88b5 bytes) 2024-09-16T13:28:44.943 INFO:teuthology.orchestra.run.smithi076.stdout:1109 trunc from 0xb149c45 to 0xcbffd2 2024-09-16T13:28:44.953 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:44.950+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:44.958 INFO:teuthology.orchestra.run.smithi076.stdout:1110 read 0xa6f78 thru 0xa7255 (0x2de bytes) 2024-09-16T13:28:44.958 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:44.974 INFO:teuthology.orchestra.run.smithi076.stdout:1112 trunc from 0xcbffd2 to 0xdb1a04e 2024-09-16T13:28:44.990 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:44.990 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:44.995 INFO:teuthology.orchestra.run.smithi076.stdout:1115 trunc from 0xdb1a04e to 0x3a13e0 2024-09-16T13:28:44.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:45.004 INFO:teuthology.orchestra.run.smithi076.stdout:1116 read 0x84e27 thru 0x91986 (0xcb60 bytes) 2024-09-16T13:28:45.021 INFO:teuthology.orchestra.run.smithi076.stdout:1120 write 0x89e713c thru 0x89f69f3 (0xf8b8 bytes) 2024-09-16T13:28:45.028 INFO:teuthology.orchestra.run.smithi076.stdout:1121 write 0x83fde4b thru 0x8405950 (0x7b06 bytes) 2024-09-16T13:28:45.034 INFO:teuthology.orchestra.run.smithi076.stdout:1122 read 0x6b05aa2 thru 0x6b0dfe6 (0x8545 bytes) 2024-09-16T13:28:45.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:45.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:28:45.039 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T13:28:45.039 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-16T13:28:45.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T13:28:45.040 INFO:teuthology.orchestra.run.smithi076.stdout:1123 writesame 0x8d29dc2 thru 0x8d30828 data_size 0x1(0x6a67 bytes) 2024-09-16T13:28:45.045 INFO:teuthology.orchestra.run.smithi076.stdout:1124 write 0x971941b thru 0x9720c5d (0x7843 bytes) 2024-09-16T13:28:45.053 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:45.051+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:45.059 INFO:teuthology.orchestra.run.smithi076.stdout:1125 write 0xc9f63f3 thru 0xca06395 (0xffa3 bytes) 2024-09-16T13:28:45.066 INFO:teuthology.orchestra.run.smithi076.stdout:1126 read 0x2adc5f1 thru 0x2ae7954 (0xb364 bytes) 2024-09-16T13:28:45.072 INFO:teuthology.orchestra.run.smithi076.stdout:1129 writesame 0xd1faf1b thru 0xd209507 data_size 0x1(0xe5ed bytes) 2024-09-16T13:28:45.072 INFO:teuthology.orchestra.run.smithi076.stdout:1131 read 0x1a11249 thru 0x1a154ab (0x4263 bytes) 2024-09-16T13:28:45.073 INFO:teuthology.orchestra.run.smithi076.stdout:1132 clone 10 order 24 su 65536 sc 4 2024-09-16T13:28:45.102 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.102 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:45.155 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:45.153+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:45.208 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.209 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:45.256 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:45.253+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:45.355 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.355 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.357 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:45.355+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:45.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:45.458 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:45.457+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:45.503 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.503 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:45.595 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:45.593+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:45.649 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.649 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:45.699 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:45.697+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:45.761 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:45.758+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:45.796 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.796 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:45.861 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:45.859+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:45.903 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:45.903 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:45.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:45.961 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:45.959+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:45.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:46.009 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:46.009 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:46.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:46.060 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T13:28:46.062 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:46.060+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:46.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:28:46.108 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:46.127 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:46.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:46.128 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone9 from 0xd209508 (overlap 0x3a13e0) to 0x0 2024-09-16T13:28:46.147 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:46.147 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:46.163 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:46.163 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:46.163 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:46.164 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:46.164 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:46.161+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:46.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:46.179 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:46.179 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:46.190 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-09-16T13:28:46.191 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.196 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:46.196 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:46.200 INFO:teuthology.orchestra.run.smithi076.stdout:1134 trunc from 0xd209508 to 0x64667a2 2024-09-16T13:28:46.215 INFO:teuthology.orchestra.run.smithi076.stdout:1135 write 0x4a9750 thru 0x4b7db5 (0xe666 bytes) 2024-09-16T13:28:46.217 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:46.217 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:46.226 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.226 INFO:teuthology.orchestra.run.smithi076.stdout:1137 write 0x2fc2e21 thru 0x2fc62e1 (0x34c1 bytes) 2024-09-16T13:28:46.240 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:46.240 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:46.243 INFO:teuthology.orchestra.run.smithi076.stdout:1138 writesame 0x7b670c5 thru 0x7b69a6e data_size 0x1(0x29aa bytes) 2024-09-16T13:28:46.250 INFO:teuthology.orchestra.run.smithi076.stdout:1139 trunc from 0x7b69a6f to 0xcee505a 2024-09-16T13:28:46.263 INFO:teuthology.orchestra.run.smithi076.stdout:1140 trunc from 0xcee505a to 0x67c7381 2024-09-16T13:28:46.263 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:46.262+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:46.282 INFO:teuthology.orchestra.run.smithi076.stdout:1142 write 0xc655caf thru 0xc65dcc4 (0x8016 bytes) 2024-09-16T13:28:46.285 INFO:teuthology.orchestra.run.smithi076.stdout:1144 write 0x6271201 thru 0x6280293 (0xf093 bytes) 2024-09-16T13:28:46.289 INFO:teuthology.orchestra.run.smithi076.stdout:1145 punch from 0xc257795 to 0xc2643bc, (0xcc27 bytes) 2024-09-16T13:28:46.289 INFO:teuthology.orchestra.run.smithi076.stdout:1147 trunc from 0xc65dcc5 to 0xc4323bc 2024-09-16T13:28:46.294 INFO:teuthology.orchestra.run.smithi076.stdout:1148 write 0x34af86a thru 0x34b4e34 (0x55cb bytes) 2024-09-16T13:28:46.296 INFO:teuthology.orchestra.run.smithi076.stdout:1149 write 0x8b68860 thru 0x8b71361 (0x8b02 bytes) 2024-09-16T13:28:46.307 INFO:teuthology.orchestra.run.smithi076.stdout:1150 write 0xec4a74f thru 0xec571b4 (0xca66 bytes) 2024-09-16T13:28:46.317 INFO:teuthology.orchestra.run.smithi076.stdout:1151 trunc from 0xec571b5 to 0xaf09466 2024-09-16T13:28:46.319 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:46.319 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:46.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:46.319 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.027355350000001, 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:46.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:46.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:28:46.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.326 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.326005152999997, 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:28:46.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325882757000002, 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.333 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:28:46.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325770368999997, 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:28:46.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.345 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325707555000001, 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:28:46.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:28:46.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325560125000003, 2024-09-16T13:28:46.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:28:46.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:46.357 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325544942000001, 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:28:46.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325523895000003, 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.363 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:28:46.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:28:46.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325436414999999, 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:28:46.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:28:46.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325413906000001, 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.375 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:28:46.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325367579000002, 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.381 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:28:46.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325200821999999, 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:28:46.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.325186057000003, 2024-09-16T13:28:46.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:28:46.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.979863739999999, 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:28:46.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.405 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.979826062000001, 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:28:46.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:28:46.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.979803249, 2024-09-16T13:28:46.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:28:46.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.979718036000001, 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:28:46.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:28:46.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 11.673139110999999, 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:28:46.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:28:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 11.647230877, 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:28:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:28:46.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 11.578465722000001, 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:28:46.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:46.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:46.438 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:46.440 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:46.438+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:46.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:46.442 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.442 INFO:teuthology.orchestra.run.smithi076.stdout:1153 read 0xd0b68 thru 0xdc9bd (0xbe56 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1154 write 0x3b62d11 thru 0x3b71242 (0xe532 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1155 punch from 0x4bf0c60 to 0x4bff7ee, (0xeb8e bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1156 read 0x656c1d6 thru 0x6573749 (0x7574 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1157 write 0x4ecd9f5 thru 0x4edb83c (0xde48 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1158 punch from 0xb0d39b to 0xb19c94, (0xc8f9 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1159 trunc from 0xaf09466 to 0x7eb4e53 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1160 trunc from 0x7eb4e53 to 0x2f1607f 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1161 read 0x99d8fb thru 0x9a1e04 (0x450a bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1162 write 0xc119085 thru 0xc1247f6 (0xb772 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1165 punch from 0x52c7405 to 0x52d086d, (0x9468 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1166 read 0x31f4ab2 thru 0x31f96a3 (0x4bf2 bytes) 2024-09-16T13:28:46.443 INFO:teuthology.orchestra.run.smithi076.stdout:1167 punch from 0x1285d5d to 0x12890ca, (0x336d bytes) 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1168 read 0x40f56d4 thru 0x40f585e (0x18b bytes) 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1170 trunc from 0xc1247f7 to 0x12e6e00 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1173 trunc from 0x12e6e00 to 0x8d11515 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1175 punch from 0x39df798 to 0x39e4988, (0x51f0 bytes) 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1176 writesame 0x234f34b thru 0x235dc44 data_size 0xb(0xe8fa bytes) 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1178 punch from 0x4efe676 to 0x4f0a55f, (0xbee9 bytes) 2024-09-16T13:28:46.444 INFO:teuthology.orchestra.run.smithi076.stdout:1179 writesame 0x6da7530 thru 0x6db1566 data_size 0xd(0xa037 bytes) 2024-09-16T13:28:46.448 INFO:teuthology.orchestra.run.smithi076.stdout:1180 read 0x20f24ae thru 0x20f4632 (0x2185 bytes) 2024-09-16T13:28:46.449 INFO:teuthology.orchestra.run.smithi076.stdout:1182 writesame 0x139ab5c thru 0x13a33c0 data_size 0x67(0x8865 bytes) 2024-09-16T13:28:46.451 INFO:teuthology.orchestra.run.smithi076.stdout:1184 write 0x72f64a5 thru 0x730629f (0xfdfb bytes) 2024-09-16T13:28:46.458 INFO:teuthology.orchestra.run.smithi076.stdout:1185 trunc from 0x8d11515 to 0xbd4ed6e 2024-09-16T13:28:46.463 INFO:teuthology.orchestra.run.smithi076.stdout:1187 write 0x858d964 thru 0x859b5b5 (0xdc52 bytes) 2024-09-16T13:28:46.471 INFO:teuthology.orchestra.run.smithi076.stdout:1188 punch from 0xb04c6fe to 0xb04f29f, (0x2ba1 bytes) 2024-09-16T13:28:46.471 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.478 INFO:teuthology.orchestra.run.smithi076.stdout:1190 write 0xdd6e957 thru 0xdd7e281 (0xf92b bytes) 2024-09-16T13:28:46.484 INFO:teuthology.orchestra.run.smithi076.stdout:1191 punch from 0x6f4208c to 0x6f43875, (0x17e9 bytes) 2024-09-16T13:28:46.484 INFO:teuthology.orchestra.run.smithi076.stdout:1192 read 0x89502c thru 0x896d4e (0x1d23 bytes) 2024-09-16T13:28:46.484 INFO:teuthology.orchestra.run.smithi076.stdout:1193 read 0x1da8497 thru 0x1db7872 (0xf3dc bytes) 2024-09-16T13:28:46.485 INFO:teuthology.orchestra.run.smithi076.stdout:1196 writesame 0x5c886e thru 0x5ccf4d data_size 0x1c(0x46e0 bytes) 2024-09-16T13:28:46.500 INFO:teuthology.orchestra.run.smithi076.stdout:1197 trunc from 0xdd7e282 to 0x2236b73 2024-09-16T13:28:46.512 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.513 INFO:teuthology.orchestra.run.smithi076.stdout:1199 trunc from 0x2236b73 to 0x1367bd9 2024-09-16T13:28:46.519 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.535 INFO:teuthology.orchestra.run.smithi076.stdout:1202 write 0x9dda80e thru 0x9de4e43 (0xa636 bytes) 2024-09-16T13:28:46.538 INFO:teuthology.orchestra.run.smithi076.stdout:1203 read 0x5d99cf8 thru 0x5d9d263 (0x356c bytes) 2024-09-16T13:28:46.538 INFO:teuthology.orchestra.run.smithi076.stdout:1204 write 0x6ecf178 thru 0x6ed1a6d (0x28f6 bytes) 2024-09-16T13:28:46.541 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:46.538+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:46.542 INFO:teuthology.orchestra.run.smithi076.stdout:1205 write 0x28e03c9 thru 0x28e88f8 (0x8530 bytes) 2024-09-16T13:28:46.544 INFO:teuthology.orchestra.run.smithi076.stdout:1207 read 0x7ada965 thru 0x7ae723f (0xc8db bytes) 2024-09-16T13:28:46.544 INFO:teuthology.orchestra.run.smithi076.stdout:1208 writesame 0x3dde86 thru 0x3df8a9 data_size 0x1c(0x1a24 bytes) 2024-09-16T13:28:46.551 INFO:teuthology.orchestra.run.smithi076.stdout:1209 writesame 0xa98725c thru 0xa98b136 data_size 0x1(0x3edb bytes) 2024-09-16T13:28:46.552 INFO:teuthology.orchestra.run.smithi076.stdout:1210 write 0x4cbfcf thru 0x4d0101 (0x4133 bytes) 2024-09-16T13:28:46.553 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:1212 read 0x234c3d thru 0x23cf72 (0x8336 bytes) 2024-09-16T13:28:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:1213 read 0x46b75e6 thru 0x46bb22c (0x3c47 bytes) 2024-09-16T13:28:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:1215 writesame 0xa32fca8 thru 0xa331d5f data_size 0x2(0x20b8 bytes) 2024-09-16T13:28:46.557 INFO:teuthology.orchestra.run.smithi076.stdout:1219 write 0x820e6c4 thru 0x8214529 (0x5e66 bytes) 2024-09-16T13:28:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:46.567 INFO:teuthology.orchestra.run.smithi076.stdout:1222 write 0xbea8d3c thru 0xbeb2f89 (0xa24e bytes) 2024-09-16T13:28:46.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:46.574 INFO:teuthology.orchestra.run.smithi076.stdout:1223 trunc from 0xbeb2f8a to 0x7bc92a0 2024-09-16T13:28:46.584 INFO:teuthology.orchestra.run.smithi076.stdout:1224 trunc from 0x7bc92a0 to 0x8380734 2024-09-16T13:28:46.589 INFO:teuthology.orchestra.run.smithi076.stdout:1226 write 0x9a417c thru 0x9a4efc (0xd81 bytes) 2024-09-16T13:28:46.590 INFO:teuthology.orchestra.run.smithi076.stdout:1228 read 0x7b8be0b thru 0x7b94e11 (0x9007 bytes) 2024-09-16T13:28:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:1229 write 0x9ba5e56 thru 0x9bad4c6 (0x7671 bytes) 2024-09-16T13:28:46.602 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.602 INFO:teuthology.orchestra.run.smithi076.stdout:1232 read 0x172a0bb thru 0x173092d (0x6873 bytes) 2024-09-16T13:28:46.602 INFO:teuthology.orchestra.run.smithi076.stdout:1233 writesame 0x4a160ee thru 0x4a1bba9 data_size 0x2(0x5abc bytes) 2024-09-16T13:28:46.617 INFO:teuthology.orchestra.run.smithi076.stdout:1234 write 0x71984e4 thru 0x719ce63 (0x4980 bytes) 2024-09-16T13:28:46.619 INFO:teuthology.orchestra.run.smithi076.stdout:1239 write 0x619f85a thru 0x61a9fcc (0xa773 bytes) 2024-09-16T13:28:46.624 INFO:teuthology.orchestra.run.smithi076.stdout:1241 writesame 0x5bc39f2 thru 0x5bcaffc data_size 0x7(0x760b bytes) 2024-09-16T13:28:46.631 INFO:teuthology.orchestra.run.smithi076.stdout:1242 writesame 0x754c8bd thru 0x7556799 data_size 0x1(0x9edd bytes) 2024-09-16T13:28:46.641 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:46.638+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:46.681 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:46.681 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:46.682 INFO:teuthology.orchestra.run.smithi076.stdout:1243 write 0xb9ad023 thru 0xb9b9178 (0xc156 bytes) 2024-09-16T13:28:46.686 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.687 INFO:teuthology.orchestra.run.smithi076.stdout:1245 writesame 0xaf88d6 thru 0xb075c6 data_size 0x1(0xecf1 bytes) 2024-09-16T13:28:46.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:46.742 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:46.740+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:46.744 INFO:teuthology.orchestra.run.smithi076.stdout:1246 punch from 0x24d4633 to 0x24d59e5, (0x13b2 bytes) 2024-09-16T13:28:46.745 INFO:teuthology.orchestra.run.smithi076.stdout:1247 read 0xa52da09 thru 0xa538ede (0xb4d6 bytes) 2024-09-16T13:28:46.750 INFO:teuthology.orchestra.run.smithi076.stdout:1249 writesame 0xc006b3e thru 0xc0143b2 data_size 0x83(0xd875 bytes) 2024-09-16T13:28:46.750 INFO:teuthology.orchestra.run.smithi076.stdout:1251 write 0x98d2544 thru 0x98dbf22 (0x99df bytes) 2024-09-16T13:28:46.754 INFO:teuthology.orchestra.run.smithi076.stdout:1252 write 0x26310c7 thru 0x2631b27 (0xa61 bytes) 2024-09-16T13:28:46.759 INFO:teuthology.orchestra.run.smithi076.stdout:1254 trunc from 0xc0143b3 to 0x952f7bd 2024-09-16T13:28:46.765 INFO:teuthology.orchestra.run.smithi076.stdout:1255 writesame 0x6480ded thru 0x6481d10 data_size 0x13(0xf24 bytes) 2024-09-16T13:28:46.766 INFO:teuthology.orchestra.run.smithi076.stdout:1256 write 0x101ecb6 thru 0x1028ffb (0xa346 bytes) 2024-09-16T13:28:46.769 INFO:teuthology.orchestra.run.smithi076.stdout:1257 read 0x847c90c thru 0x84888cf (0xbfc4 bytes) 2024-09-16T13:28:46.772 INFO:teuthology.orchestra.run.smithi076.stdout:1261 trunc from 0x952f7bd to 0x7493176 2024-09-16T13:28:46.783 INFO:teuthology.orchestra.run.smithi076.stdout:1262 write 0x4dc6b00 thru 0x4dd0943 (0x9e44 bytes) 2024-09-16T13:28:46.794 INFO:teuthology.orchestra.run.smithi076.stdout:1263 write 0x9827b99 thru 0x9834037 (0xc49f bytes) 2024-09-16T13:28:46.799 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:46.799 INFO:teuthology.orchestra.run.smithi076.stdout:1268 read 0xd4d883 thru 0xd509c6 (0x3144 bytes) 2024-09-16T13:28:46.800 INFO:teuthology.orchestra.run.smithi076.stdout:1269 read 0x9135392 thru 0x914330e (0xdf7d bytes) 2024-09-16T13:28:46.800 INFO:teuthology.orchestra.run.smithi076.stdout:1270 writesame 0x4b17491 thru 0x4b207ab data_size 0x1(0x931b bytes) 2024-09-16T13:28:46.837 INFO:teuthology.orchestra.run.smithi076.stdout:1273 punch from 0x745dd22 to 0x746a770, (0xca4e bytes) 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6737661409999998, 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:28:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:28:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:28:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6682905760000004, 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:28:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:28:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:28:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.9079637690000002, 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:28:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:28:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.8914256439999999, 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:28:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:28:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8144 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3538944~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.548862261, 2024-09-16T13:28:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041437515000000001, 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8144 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290569+0000", 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-07 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290571+0000", 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9120000000000001e-06 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290578+0000", 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9779999999999999e-06 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290582+0000", 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.331618+0000", 2024-09-16T13:28:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041035335999999999 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.331688+0000", 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0128000000000003e-05 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332007+0000", 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031891500000000002 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8145 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5488450749999991, 2024-09-16T13:28:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041847961000000003, 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8145 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290587+0000", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.03e-07 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290589+0000", 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.086e-06 2024-09-16T13:28:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290596+0000", 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1509999999999998e-06 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290600+0000", 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.692e-06 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332033+0000", 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041433316999999997 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332120+0000", 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6847000000000004e-05 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332434+0000", 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031446499999999999 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8146 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5488281149999992, 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042182996, 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8146 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:28:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290604+0000", 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290605+0000", 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:28:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290612+0000", 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.686e-06 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290616+0000", 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332462+0000", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041846176999999998 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332534+0000", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2398999999999996e-05 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332786+0000", 2024-09-16T13:28:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025215300000000002 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8147 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5488120429999999, 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042474660999999997, 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8147 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:28:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290620+0000", 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290621+0000", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290627+0000", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9020000000000004e-06 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290632+0000", 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6609999999999999e-06 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332812+0000", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042180278000000002 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332854+0000", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1857999999999998e-05 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333094+0000", 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000239905 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5487957619999992, 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:28:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5487787839999996, 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.548762194, 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5487398490000004, 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5487218689999995, 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:28:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:28:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:28:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5487060649999993, 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:28:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5486905039999996, 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:28:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:28:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5486727899999995, 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.5486462070000009, 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:28:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:28:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.7070922939999997, 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:28:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.5096527230000003, 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:28:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:28:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4991539759999997, 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:28:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:46.949 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:46.951 INFO:teuthology.orchestra.run.smithi076.stdout:1274 write 0xa864d23 thru 0xa87417d (0xf45b bytes) 2024-09-16T13:28:46.951 INFO:teuthology.orchestra.run.smithi076.stdout:1275 punch from 0x177f9b7 to 0x1785b86, (0x61cf bytes) 2024-09-16T13:28:46.951 INFO:teuthology.orchestra.run.smithi076.stdout:1276 punch from 0x5e63b91 to 0x5e71c48, (0xe0b7 bytes) 2024-09-16T13:28:46.951 INFO:teuthology.orchestra.run.smithi076.stdout:1277 clone 11 order 25 su 2097152 sc 4 2024-09-16T13:28:46.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:46.953 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:46.948+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:47.051 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:47.049+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:47.058 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25827 4.4 4:2249ef24:::rbd_object_map.11a98415fd69.000000000000000c:head [call rbd.object_map_load in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:47.099028+0000", 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.5748999999999997e-05, 2024-09-16T13:28:47.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6582999999999994e-05, 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25827 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099028+0000", 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099028+0000", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099029+0000", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3900000000000004e-07 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099033+0000", 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2300000000000002e-06 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099045+0000", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1338999999999999e-05 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099051+0000", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:47.099087+0000", 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5617999999999997e-05 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.104 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:28:47.104 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 1 2024-09-16T13:28:47.104 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:47.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:47.125 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone10 from 0xa87417e (overlap 0x12e6e00) to 0xd3b366 2024-09-16T13:28:47.167 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:47.165+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:47.203 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-09-16T13:28:47.204 INFO:teuthology.orchestra.run.smithi076.stdout:1279 write 0x11c44a0 thru 0x11c73dc (0x2f3d bytes) 2024-09-16T13:28:47.229 INFO:teuthology.orchestra.run.smithi076.stdout:1280 read 0x9d370bc thru 0x9d401b4 (0x90f9 bytes) 2024-09-16T13:28:47.229 INFO:teuthology.orchestra.run.smithi076.stdout:1281 read 0x5f991b thru 0x5fe62d (0x4d13 bytes) 2024-09-16T13:28:47.230 INFO:teuthology.orchestra.run.smithi076.stdout:1282 writesame 0x2d016a0 thru 0x2d0a4b3 data_size 0x15(0x8e14 bytes) 2024-09-16T13:28:47.237 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.237 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:47.237 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:47.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:47.237 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:47.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:47.248 INFO:teuthology.orchestra.run.smithi076.stdout:1283 write 0xbff51bf thru 0xbff5fdb (0xe1d bytes) 2024-09-16T13:28:47.252 INFO:teuthology.orchestra.run.smithi076.stdout:1284 trunc from 0xbff5fdc to 0xbf8acb8 2024-09-16T13:28:47.252 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:47.250+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:47.258 INFO:teuthology.orchestra.run.smithi076.stdout:1285 trunc from 0xbf8acb8 to 0xc2c855d 2024-09-16T13:28:47.269 INFO:teuthology.orchestra.run.smithi076.stdout:1286 write 0xd27281f thru 0xd281f04 (0xf6e6 bytes) 2024-09-16T13:28:47.273 INFO:teuthology.orchestra.run.smithi076.stdout:1288 read 0x3a046ff thru 0x3a09758 (0x505a bytes) 2024-09-16T13:28:47.274 INFO:teuthology.orchestra.run.smithi076.stdout:1291 write 0x208997d thru 0x208e459 (0x4add bytes) 2024-09-16T13:28:47.276 INFO:teuthology.orchestra.run.smithi076.stdout:1292 read 0x9dbed52 thru 0x9dcc4aa (0xd759 bytes) 2024-09-16T13:28:47.276 INFO:teuthology.orchestra.run.smithi076.stdout:1295 writesame 0xace669a thru 0xace7358 data_size 0x1(0xcbf bytes) 2024-09-16T13:28:47.281 INFO:teuthology.orchestra.run.smithi076.stdout:1296 write 0x5e663ec thru 0x5e713a0 (0xafb5 bytes) 2024-09-16T13:28:47.297 INFO:teuthology.orchestra.run.smithi076.stdout:1297 trunc from 0xd281f05 to 0x6ff890b 2024-09-16T13:28:47.315 INFO:teuthology.orchestra.run.smithi076.stdout:1298 clone 12 order 23 su 262144 sc 4 2024-09-16T13:28:47.353 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:47.351+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.915096281, 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:1146 4.f e37/32 ondisk, result = 0)", 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.912130181, 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014045078000000001, 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445311+0000", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6499999999999998e-07 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445315+0000", 2024-09-16T13:28:47.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0319999999999997e-06 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445327+0000", 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2495e-05 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445333+0000", 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9449999999999998e-06 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459267+0000", 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013933555 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459277+0000", 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0594000000000001e-05 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459355+0000", 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7591999999999998e-05 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.742697105, 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3373 4.b e38/32 4:d151a9d0:::rbd_data.11a976a8123d.000000000000000a:head v 38'14, mlcod=38'11)", 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.651393801999999, 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014151797000000001, 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706047+0000", 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706592+0000", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054438299999999995 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706613+0000", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1118999999999999e-05 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706620+0000", 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706670+0000", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280999999999998e-05 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706697+0000", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7452000000000001e-05 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720145+0000", 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013447506999999999 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720198+0000", 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3232999999999997e-05 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.5076252310000005, 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:28:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:28:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:28:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.6549251090000006, 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:47.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:28:47.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:28:47.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.6338612010000002, 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.3297881629999999, 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2251943409999999, 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.398 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:28:47.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:28:47.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:28:47.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21707 4.14 e46/32 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head v 46'74, mlcod=45'72)", 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:47.402 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2204511079999998, 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015037457000000001, 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000003e-07 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137531+0000", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054107700000000005 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137543+0000", 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2156e-05 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137548+0000", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.463e-06 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137896+0000", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034767700000000001 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137917+0000", 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1523999999999999e-05 2024-09-16T13:28:47.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.151679+0000", 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013761997 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152026+0000", 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034681000000000002 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2186507000000004, 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.410 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23819 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5417496389999998, 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018047839999999999, 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23819 2024-09-16T13:28:47.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815691+0000", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.054e-06 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815696+0000", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1749999999999996e-06 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815708+0000", 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1446e-05 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815715+0000", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824348+0000", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086330599999999997 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824416+0000", 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8213999999999994e-05 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824679+0000", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026342200000000003 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826830+0000", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002150749 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833601+0000", 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067705100000000004 2024-09-16T13:28:47.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833698+0000", 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7188000000000005e-05 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833738+0000", 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0173000000000003e-05 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23820 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5417189850000002, 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018255289000000001, 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23820 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815725+0000", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7840000000000001e-06 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815732+0000", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9040000000000003e-06 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815737+0000", 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5549999999999996e-06 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825011+0000", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092747530000000002 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825098+0000", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6287999999999998e-05 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825475+0000", 2024-09-16T13:28:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037692200000000002 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826843+0000", 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001368403 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833748+0000", 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069050989999999996 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833884+0000", 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013585 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833976+0000", 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2219999999999995e-05 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.422 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23821 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5416975920000002, 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018354471000000001, 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23821 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815743+0000", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.46e-07 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815747+0000", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889999999999999e-06 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815756+0000", 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-06 2024-09-16T13:28:47.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815760+0000", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825875+0000", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010114911000000001 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825919+0000", 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3572000000000002e-05 2024-09-16T13:28:47.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826099+0000", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018034099999999999 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827140+0000", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010402899999999999 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833988+0000", 2024-09-16T13:28:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068483349999999997 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834065+0000", 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7305999999999995e-05 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834097+0000", 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1492000000000003e-05 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25655 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.75576028399999995, 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014517317, 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25655 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601681+0000", 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6169999999999999e-06 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601688+0000", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.294e-06 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601703+0000", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4124999999999999e-05 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601709+0000", 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1090000000000003e-06 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601751+0000", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2293000000000003e-05 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601810+0000", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8786000000000003e-05 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.602757+0000", 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094687500000000002 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616128+0000", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013370885000000001 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616135+0000", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9300000000000003e-06 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616171+0000", 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5196e-05 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616197+0000", 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.72689094099999996, 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:47.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:28:47.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:28:47.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:28:47.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.71801436299999999, 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:28:47.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.55802015299999996, 2024-09-16T13:28:47.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:28:47.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.55798712399999995, 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.503 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:28:47.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:28:47.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:28:47.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.54662323000000002, 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:28:47.508 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:28:47.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.512 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:47.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:47.514 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:47.510+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:47.613 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:47.611+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:47.653 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.653 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:47.653 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:47.653 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:47.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:47.746 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:47.743+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:47.799 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.800 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:47.800 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:47.800 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:47.800 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:47.808 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:47.815 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:47.813+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:47.916 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:47.913+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:1858 4.d e38/32 4:b4b3857b:::rbd_data.11a962dad130.000000000000001d:head v 38'15, mlcod=37'13)", 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.872179741, 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006645443, 2024-09-16T13:28:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073918+0000", 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.018e-06 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075152+0000", 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012342570000000001 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075168+0000", 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5841999999999998e-05 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075174+0000", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3539999999999996e-06 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075208+0000", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3887000000000003e-05 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075221+0000", 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2845e-05 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.080526+0000", 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053053010000000001 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.080562+0000", 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5939000000000003e-05 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1932 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.868917385, 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079948060000000001, 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1932 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077180+0000", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077480+0000", 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029970499999999999 2024-09-16T13:28:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077495+0000", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5277999999999998e-05 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077501+0000", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9070000000000004e-06 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077556+0000", 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5189999999999998e-05 2024-09-16T13:28:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077635+0000", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8985000000000004e-05 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077860+0000", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000224887 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084483+0000", 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066231550000000004 2024-09-16T13:28:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085091+0000", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060761599999999995 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085139+0000", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.812e-05 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085174+0000", 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4567000000000003e-05 2024-09-16T13:28:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1933 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.868585587, 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088826889999999992, 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1933 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T13:28:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077514+0000", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077526+0000", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1834000000000001e-05 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077529+0000", 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.76e-06 2024-09-16T13:28:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083235+0000", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057052270000000002 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083314+0000", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9680999999999998e-05 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083493+0000", 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017889199999999999 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084872+0000", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013790250000000001 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086316+0000", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014437289999999999 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086358+0000", 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1962999999999998e-05 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086393+0000", 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5657000000000001e-05 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2074 4.d e38/32 4:b0be95a4:::rbd_data.11a962dad130.000000000000002a:head v 38'19, mlcod=38'17)", 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.849907905, 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071711229999999997, 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096190+0000", 2024-09-16T13:28:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1170000000000001e-06 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096783+0000", 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059312600000000003 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096792+0000", 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5689999999999994e-06 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096798+0000", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3170000000000001e-06 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096848+0000", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0818e-05 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096874+0000", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.103325+0000", 2024-09-16T13:28:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064508050000000004 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.103360+0000", 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5051e-05 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2075 4.d e38/32 4:b0be95a4:::rbd_data.11a962dad130.000000000000002a:head v 38'21, mlcod=38'17)", 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.847061399999999, 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051515320000000003, 2024-09-16T13:28:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099036+0000", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1699999999999997e-07 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099043+0000", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7449999999999995e-06 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099053+0000", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.815e-06 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099057+0000", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8999999999999999e-06 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.101217+0000", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021601839999999999 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.101240+0000", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2770000000000001e-05 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.104159+0000", 2024-09-16T13:28:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002919081 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.104186+0000", 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7319999999999999e-05 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2319 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.829374353, 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080751409999999992, 2024-09-16T13:28:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2319 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116724+0000", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1619999999999998e-06 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116732+0000", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.554e-06 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116751+0000", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9454999999999999e-05 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116758+0000", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0650000000000001e-06 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116832+0000", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4058000000000001e-05 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116965+0000", 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013282999999999999 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124797+0000", 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078320170000000001 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.814377984, 2024-09-16T13:28:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 38/32 rep_tid 220 entries 38'14 (0'0) error 4:9f82cb54:::rbd_data.11a962dad130.0000000000000049:head by client.4521.0:2478 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 37'12)", 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.804369202, 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051641090000000001, 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5699999999999999e-07 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141734+0000", 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4529999999999999e-06 2024-09-16T13:28:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141749+0000", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4749999999999999e-05 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141755+0000", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3330000000000002e-06 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141816+0000", 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1124999999999995e-05 2024-09-16T13:28:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146891+0000", 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050751909999999997 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2615 4.4 e38/32 4:236e8c22:::rbd_data.11a962dad130.0000000000000047:head v 38'17, mlcod=37'15)", 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.784988887000001, 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054677249999999997, 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161108+0000", 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161204+0000", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6316000000000004e-05 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161214+0000", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0176e-05 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161219+0000", 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2024-09-16T13:28:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161363+0000", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014346300000000001 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161381+0000", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7706e-05 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166538+0000", 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051578750000000001 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166575+0000", 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6823999999999998e-05 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2694 4.d e38/32 4:b2b91969:::rbd_data.11a962dad130.000000000000005c:head v 38'27, mlcod=38'24)", 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.777722388000001, 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005599026, 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168375+0000", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000001e-07 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168385+0000", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0088000000000001e-05 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168400+0000", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4866e-05 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168405+0000", 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8159999999999999e-06 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168443+0000", 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7724e-05 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168461+0000", 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7989999999999999e-05 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.173938+0000", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054768960000000002 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.173973+0000", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4906000000000001e-05 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2814 4.1c 4:3b21c817:::rbd_data.11a962dad130.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.753012612999999, 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052003220000000003, 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2814 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193086+0000", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193364+0000", 2024-09-16T13:28:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027807999999999999 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193381+0000", 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949999999999999e-05 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193389+0000", 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193435+0000", 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5717000000000003e-05 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193562+0000", 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012698900000000001 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193796+0000", 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000233434 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198030+0000", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042348960000000001 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198216+0000", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018563 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198249+0000", 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3123000000000001e-05 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198284+0000", 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4919999999999998e-05 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2815 4.1c 4:3b21c817:::rbd_data.11a962dad130.0000000000000010:head [truncate 196608] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.752696189, 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055168279999999997, 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2815 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193401+0000", 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193403+0000", 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7429999999999998e-06 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193410+0000", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3169999999999998e-06 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193414+0000", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3540000000000002e-06 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197099+0000", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036846460000000002 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197160+0000", 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0717000000000002e-05 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197286+0000", 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126799 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198454+0000", 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001167403 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198869+0000", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00041520000000000001 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198896+0000", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7316000000000001e-05 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198917+0000", 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0772999999999999e-05 2024-09-16T13:28:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:7508 4.19 e41/32 4:9af47f66:::rbd_data.11a91a08d9ca.0000000000000012:head v 41'26, mlcod=41'23)", 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.386008254, 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065137279999999999, 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560089+0000", 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0410000000000001e-06 2024-09-16T13:28:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560512+0000", 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000422619 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560529+0000", 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7592999999999999e-05 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560536+0000", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560580+0000", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560602+0000", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1933000000000002e-05 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.566558+0000", 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059559110000000004 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.566602+0000", 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3581999999999997e-05 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 46/32 rep_tid 455 entries 46'49 (0'0) error 4:9f62d19d:::rbd_data.11a98415fd69.0000000000000034:head by client.4521.0:21931 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 46'47)", 2024-09-16T13:28:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7801429979999996, 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076044759999999998, 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165954+0000", 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8999999999999996e-07 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165961+0000", 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9079999999999998e-06 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165974+0000", 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2874000000000001e-05 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165982+0000", 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166056+0000", 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4771000000000006e-05 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173558+0000", 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075015560000000004 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21932 4.19 e46/32 4:989741d8:::rbd_data.11a98415fd69.0000000000000030:head v 46'50, mlcod=46'47)", 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7797677719999996, 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072626100000000001, 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166335+0000", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6470000000000003e-06 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166348+0000", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599e-05 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166354+0000", 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1270000000000002e-06 2024-09-16T13:28:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166402+0000", 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8764999999999998e-05 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166419+0000", 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6237000000000001e-05 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173562+0000", 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007143445 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173591+0000", 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9135000000000001e-05 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 46/32 rep_tid 458 entries 46'51 (0'0) error 4:9c2347db:::rbd_data.11a98415fd69.0000000000000037:head by client.4521.0:21939 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 46'47)", 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7790086949999999, 2024-09-16T13:28:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006512866, 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167097+0000", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2619999999999998e-06 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167115+0000", 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7849000000000001e-05 2024-09-16T13:28:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167123+0000", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0690000000000002e-06 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167178+0000", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5825000000000002e-05 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173601+0000", 2024-09-16T13:28:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064220969999999999 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21984 4.19 e46/32 4:9f82c0fa:::rbd_data.11a98415fd69.0000000000000039:head v 46'55, mlcod=46'51)", 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7693135619999998, 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066286280000000001, 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3e-07 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176791+0000", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1210000000000001e-06 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176803+0000", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2571e-05 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176809+0000", 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:28:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176865+0000", 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5828000000000001e-05 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176881+0000", 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6359e-05 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.183355+0000", 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064736849999999999 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.183411+0000", 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6823999999999997e-05 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:22015 4.4 e46/32 4:2249ef24:::rbd_object_map.11a98415fd69.000000000000000c:head v 46'57, mlcod=46'55)", 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7664803459999998, 2024-09-16T13:28:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078776269999999999, 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179617+0000", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2500000000000004e-07 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179624+0000", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.328e-06 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179636+0000", 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1659e-05 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179642+0000", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8529999999999997e-06 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179679+0000", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6977000000000002e-05 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179699+0000", 2024-09-16T13:28:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0630999999999999e-05 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187421+0000", 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077212310000000003 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187494+0000", 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3022999999999995e-05 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:22062 4.1c 4:3e94e98f:::rbd_data.11a98415fd69.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.7641639869999999, 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056513229999999998, 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 22062 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181934+0000", 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5549999999999999e-06 2024-09-16T13:28:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181940+0000", 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3230000000000003e-06 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181955+0000", 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4859e-05 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181962+0000", 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5259999999999999e-06 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182007+0000", 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5351e-05 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182085+0000", 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7828000000000001e-05 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182284+0000", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198844 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187425+0000", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051414800000000004 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187501+0000", 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6264e-05 2024-09-16T13:28:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187549+0000", 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7729000000000003e-05 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187584+0000", 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4564000000000003e-05 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:23483 4.d e46/32 4:b36208b4:::rbd_data.11a9ff22d86c.0000000000000034:head v 46'50, mlcod=46'48)", 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.1653568429999996, 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061942059999999998, 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780741+0000", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1179999999999999e-06 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781581+0000", 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084065599999999996 2024-09-16T13:28:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781600+0000", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8430000000000001e-05 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781607+0000", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1840000000000002e-06 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781654+0000", 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6737999999999999e-05 2024-09-16T13:28:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781678+0000", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.438e-05 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.786888+0000", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005210279 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.786934+0000", 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5420999999999998e-05 2024-09-16T13:28:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:48.040 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:48.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:48.042 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:48.039+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:48.138 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone11 from 0x6ff890b (overlap 0x6ff890b) to 0x1664e9b 2024-09-16T13:28:48.141 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:48.140+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:48.198 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:48.198 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:48.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:48.237 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-09-16T13:28:48.243 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:48.240+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:48.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:48.257 INFO:teuthology.orchestra.run.smithi076.stdout:1301 write 0x1f06a05 thru 0x1f15f67 (0xf563 bytes) 2024-09-16T13:28:48.272 INFO:teuthology.orchestra.run.smithi076.stdout:1303 read 0x5667a33 thru 0x5673440 (0xba0e bytes) 2024-09-16T13:28:48.287 INFO:teuthology.orchestra.run.smithi076.stdout:1304 writesame 0xe6ba9a0 thru 0xe6c3358 data_size 0x1(0x89b9 bytes) 2024-09-16T13:28:48.287 INFO:teuthology.orchestra.run.smithi076.stdout:1309 writesame 0x9b9c88b thru 0x9ba456a data_size 0x18(0x7ce0 bytes) 2024-09-16T13:28:48.290 INFO:teuthology.orchestra.run.smithi076.stdout:1310 writesame 0x6929c70 thru 0x6934a0b data_size 0x52(0xad9c bytes) 2024-09-16T13:28:48.295 INFO:teuthology.orchestra.run.smithi076.stdout:1312 write 0x4e0f993 thru 0x4e1bbc0 (0xc22e bytes) 2024-09-16T13:28:48.308 INFO:teuthology.orchestra.run.smithi076.stdout:1313 trunc from 0xe6c3359 to 0x8c47152 2024-09-16T13:28:48.315 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.315 INFO:teuthology.orchestra.run.smithi076.stdout:1316 read 0x1298bfd thru 0x12a62c3 (0xd6c7 bytes) 2024-09-16T13:28:48.315 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.315 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.315 INFO:teuthology.orchestra.run.smithi076.stdout:1320 read 0x6fd0d54 thru 0x6fdfbe9 (0xee96 bytes) 2024-09-16T13:28:48.316 INFO:teuthology.orchestra.run.smithi076.stdout:1321 read 0x725c3c0 thru 0x7267e1a (0xba5b bytes) 2024-09-16T13:28:48.316 INFO:teuthology.orchestra.run.smithi076.stdout:1322 read 0x5ae20d thru 0x5b4dbb (0x6baf bytes) 2024-09-16T13:28:48.316 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:48.316 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:48.316 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:48.317 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:48.317 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:48.325 INFO:teuthology.orchestra.run.smithi076.stdout:1323 trunc from 0x8c47152 to 0x123115e 2024-09-16T13:28:48.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:48.371 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:48.368+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:48.375 INFO:teuthology.orchestra.run.smithi076.stdout:1324 writesame 0xebf0f55 thru 0xebf3692 data_size 0x1(0x273e bytes) 2024-09-16T13:28:48.376 INFO:teuthology.orchestra.run.smithi076.stdout:1326 trunc from 0xebf3693 to 0xe03b5b6 2024-09-16T13:28:48.382 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.382 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.382 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.382 INFO:teuthology.orchestra.run.smithi076.stdout:1334 punch from 0x5f1a8cd to 0x5f1fec8, (0x55fb bytes) 2024-09-16T13:28:48.383 INFO:teuthology.orchestra.run.smithi076.stdout:1336 write 0x2354913 thru 0x2356114 (0x1802 bytes) 2024-09-16T13:28:48.385 INFO:teuthology.orchestra.run.smithi076.stdout:1337 writesame 0xd588939 thru 0xd59415c data_size 0xa(0xb824 bytes) 2024-09-16T13:28:48.390 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:48.390 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:48.395 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.395 INFO:teuthology.orchestra.run.smithi076.stdout:1339 write 0xacada45 thru 0xacbd9cd (0xff89 bytes) 2024-09-16T13:28:48.401 INFO:teuthology.orchestra.run.smithi076.stdout:1340 read 0x8949e3f thru 0x8951941 (0x7b03 bytes) 2024-09-16T13:28:48.401 INFO:teuthology.orchestra.run.smithi076.stdout:1341 read 0xef030e thru 0xefccfe (0xc9f1 bytes) 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:1343 read 0x8a66c03 thru 0x8a738ed (0xcceb bytes) 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:1345 read 0xc950bf3 thru 0xc95a250 (0x965e bytes) 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:1347 read 0xa50cb05 thru 0xa511f9a (0x5496 bytes) 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:1348 read 0x3830777 thru 0x3833fd1 (0x385b bytes) 2024-09-16T13:28:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:1349 writesame 0xd3ad72c thru 0xd3aff43 data_size 0x4(0x2818 bytes) 2024-09-16T13:28:48.403 INFO:teuthology.orchestra.run.smithi076.stdout:1350 read 0xcf0d3f0 thru 0xcf15f24 (0x8b35 bytes) 2024-09-16T13:28:48.403 INFO:teuthology.orchestra.run.smithi076.stdout:1351 read 0xb4e951e thru 0xb4f4ad7 (0xb5ba bytes) 2024-09-16T13:28:48.403 INFO:teuthology.orchestra.run.smithi076.stdout:1352 writesame 0xa949438 thru 0xa9533cb data_size 0xe(0x9f94 bytes) 2024-09-16T13:28:48.406 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:48.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:48.407 INFO:teuthology.orchestra.run.smithi076.stdout:1353 write 0xbc3e3d1 thru 0xbc43aa5 (0x56d5 bytes) 2024-09-16T13:28:48.410 INFO:teuthology.orchestra.run.smithi076.stdout:1354 write 0x3801ade thru 0x3805819 (0x3d3c bytes) 2024-09-16T13:28:48.422 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:48.423 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:48.428 INFO:teuthology.orchestra.run.smithi076.stdout:1355 trunc from 0xe03b5b6 to 0x1f2d562 2024-09-16T13:28:48.439 INFO:teuthology.orchestra.run.smithi076.stdout:1356 read 0x11eb958 thru 0x11f8117 (0xc7c0 bytes) 2024-09-16T13:28:48.439 INFO:teuthology.orchestra.run.smithi076.stdout:1357 punch from 0x14320fb to 0x143f10e, (0xd013 bytes) 2024-09-16T13:28:48.441 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:48.441 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:48.444 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:48.442+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:48.458 INFO:teuthology.orchestra.run.smithi076.stdout:1359 write 0xbd4e83d thru 0xbd5be0d (0xd5d1 bytes) 2024-09-16T13:28:48.462 INFO:teuthology.orchestra.run.smithi076.stdout:1360 read 0xb5dbb62 thru 0xb5eae65 (0xf304 bytes) 2024-09-16T13:28:48.462 INFO:teuthology.orchestra.run.smithi076.stdout:1361 write 0x1c72f0d thru 0x1c7e1d1 (0xb2c5 bytes) 2024-09-16T13:28:48.465 INFO:teuthology.orchestra.run.smithi076.stdout:1362 write 0x7d163a1 thru 0x7d24d19 (0xe979 bytes) 2024-09-16T13:28:48.465 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:48.482 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:48.483 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:48.484 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:48.484 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:48.484 INFO:teuthology.orchestra.run.smithi076.stdout:1363 trunc from 0xbd5be0e to 0xd4259a 2024-09-16T13:28:48.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:48.502 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:48.502 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:48.508 INFO:teuthology.orchestra.run.smithi076.stdout:1364 write 0x8007310 thru 0x8007ec4 (0xbb5 bytes) 2024-09-16T13:28:48.512 INFO:teuthology.orchestra.run.smithi076.stdout:1365 write 0x1a177b0 thru 0x1a1dd47 (0x6598 bytes) 2024-09-16T13:28:48.514 INFO:teuthology.orchestra.run.smithi076.stdout:1366 write 0x4f5568d thru 0x4f5aafc (0x5470 bytes) 2024-09-16T13:28:48.517 INFO:teuthology.orchestra.run.smithi076.stdout:1369 read 0x9168a thru 0x91c9d (0x614 bytes) 2024-09-16T13:28:48.518 INFO:teuthology.orchestra.run.smithi076.stdout:1370 read 0x1f3da2a thru 0x1f4ce51 (0xf428 bytes) 2024-09-16T13:28:48.518 INFO:teuthology.orchestra.run.smithi076.stdout:1371 writesame 0x489125b thru 0x489fd72 data_size 0x4(0xeb18 bytes) 2024-09-16T13:28:48.519 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:48.519 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:48.545 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:48.543+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:48.547 INFO:teuthology.orchestra.run.smithi076.stdout:1372 write 0x5059eb5 thru 0x505af08 (0x1054 bytes) 2024-09-16T13:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout:1376 punch from 0x6be9411 to 0x6bf1a43, (0x8632 bytes) 2024-09-16T13:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout:1378 trunc from 0x8007ec5 to 0x532ed94 2024-09-16T13:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout:1379 punch from 0x2da0b07 to 0x2da3f69, (0x3462 bytes) 2024-09-16T13:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout:1380 punch from 0x4090918 to 0x4097082, (0x676a bytes) 2024-09-16T13:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout:1381 read 0x43fab57 thru 0x43fca29 (0x1ed3 bytes) 2024-09-16T13:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout:1382 write 0x50872e2 thru 0x508a9e4 (0x3703 bytes) 2024-09-16T13:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout:1383 read 0x1ed716f thru 0x1edf7b5 (0x8647 bytes) 2024-09-16T13:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout:1385 read 0x1c7018 thru 0x1cd1d2 (0x61bb bytes) 2024-09-16T13:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout:1386 write 0x3e415f1 thru 0x3e48f3b (0x794b bytes) 2024-09-16T13:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:1388 trunc from 0x532ed94 to 0xd5806d2 2024-09-16T13:28:48.585 INFO:teuthology.orchestra.run.smithi076.stdout:1389 writesame 0x8214dc thru 0x828380 data_size 0x37(0x6ea5 bytes) 2024-09-16T13:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout:1391 write 0xb73794b thru 0xb741b1c (0xa1d2 bytes) 2024-09-16T13:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout:1393 read 0x8258312 thru 0x825a3b8 (0x20a7 bytes) 2024-09-16T13:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout:1395 read 0x586de7 thru 0x592ec1 (0xc0db bytes) 2024-09-16T13:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout:1396 punch from 0xbf723b1 to 0xbf7fcad, (0xd8fc bytes) 2024-09-16T13:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout:1397 read 0x3236164 thru 0x3238b9c (0x2a39 bytes) 2024-09-16T13:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout:1398 read 0x4919a4a thru 0x492715c (0xd713 bytes) 2024-09-16T13:28:48.595 INFO:teuthology.orchestra.run.smithi076.stdout:1399 write 0x2ca7210 thru 0x2cacfca (0x5dbb bytes) 2024-09-16T13:28:48.598 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:48.598 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout:1400 write 0x92991b6 thru 0x92a2f74 (0x9dbf bytes) 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:1401 punch from 0x27b2f9c to 0x27c104e, (0xe0b2 bytes) 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:1405 punch from 0x8c2820 to 0x8c8da6, (0x6586 bytes) 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:1406 read 0x91e6a96 thru 0x91f494f (0xdeba bytes) 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:1407 punch from 0x53e5c1c to 0x53f053f, (0xa923 bytes) 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout:1410 writesame 0x660e7c7 thru 0x6614584 data_size 0x47(0x5dbe bytes) 2024-09-16T13:28:48.605 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout:1413 read 0x69744af thru 0x697ecf0 (0xa842 bytes) 2024-09-16T13:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout:1416 write 0x6aaa4ab thru 0x6aaa6d2 (0x228 bytes) 2024-09-16T13:28:48.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:48.609 INFO:teuthology.orchestra.run.smithi076.stdout:1417 writesame 0x8633c51 thru 0x8636175 data_size 0x1(0x2525 bytes) 2024-09-16T13:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout:1418 write 0x346aae9 thru 0x346ee4e (0x4366 bytes) 2024-09-16T13:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout:1419 read 0x83627fd thru 0x8371baf (0xf3b3 bytes) 2024-09-16T13:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout:1420 read 0x4df7252 thru 0x4e054f5 (0xe2a4 bytes) 2024-09-16T13:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout:1425 read 0xb1be34e thru 0xb1be870 (0x523 bytes) 2024-09-16T13:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout:1426 write 0xaa41d3b thru 0xaa42df0 (0x10b6 bytes) 2024-09-16T13:28:48.627 INFO:teuthology.orchestra.run.smithi076.stdout:1427 writesame 0x49c383 thru 0x4a7f61 data_size 0x1(0xbbdf bytes) 2024-09-16T13:28:48.649 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:48.647+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:48.710 INFO:teuthology.orchestra.run.smithi076.stdout:1428 read 0x6180e52 thru 0x6189ded (0x8f9c bytes) 2024-09-16T13:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout:1429 write 0x2a5ce73 thru 0x2a6ad56 (0xdee4 bytes) 2024-09-16T13:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout:1434 writesame 0x5ea5c4 thru 0x5f2a56 data_size 0x1b(0x8493 bytes) 2024-09-16T13:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout:1435 writesame 0x9d73d9a thru 0x9d82c56 data_size 0x1(0xeebd bytes) 2024-09-16T13:28:48.746 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:48.743+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:48.758 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:48.759 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:48.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout:1437 write 0x152e0f7 thru 0x1538e10 (0xad1a bytes) 2024-09-16T13:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout:1438 write 0x8af1d52 thru 0x8afc9d4 (0xac83 bytes) 2024-09-16T13:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout:1439 trunc from 0xd5806d2 to 0x2905870 2024-09-16T13:28:48.847 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:48.845+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout:1440 write 0x7b4df11 thru 0x7b5d03b (0xf12b bytes) 2024-09-16T13:28:48.858 INFO:teuthology.orchestra.run.smithi076.stdout:1442 read 0x24c6278 thru 0x24d1b54 (0xb8dd bytes) 2024-09-16T13:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout:1444 read 0x9fd625 thru 0xa05900 (0x82dc bytes) 2024-09-16T13:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout:1446 read 0x10b0358 thru 0x10b405e (0x3d07 bytes) 2024-09-16T13:28:48.868 INFO:teuthology.orchestra.run.smithi076.stdout:1447 write 0xbad1035 thru 0xbad8f2a (0x7ef6 bytes) 2024-09-16T13:28:48.872 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:48.872 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout:1450 punch from 0x17ff3dc to 0x1802acc, (0x36f0 bytes) 2024-09-16T13:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout:1452 read 0x934eaf8 thru 0x935922a (0xa733 bytes) 2024-09-16T13:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout:1454 write 0x5e02877 thru 0x5e0ae61 (0x85eb bytes) 2024-09-16T13:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout:1456 write 0x58b42c thru 0x58d457 (0x202c bytes) 2024-09-16T13:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout:1457 read 0x94b56a0 thru 0x94be3fd (0x8d5e bytes) 2024-09-16T13:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout:1458 write 0x7ad2900 thru 0x7ad43c1 (0x1ac2 bytes) 2024-09-16T13:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout:1459 writesame 0x95abbd5 thru 0x95b4bcb data_size 0x87(0x8ff7 bytes) 2024-09-16T13:28:48.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout:1460 writesame 0x6754871 thru 0x675cd5a data_size 0x6b(0x84ea bytes) 2024-09-16T13:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout:1461 writesame 0xdb7d79b thru 0xdb82999 data_size 0x1(0x51ff bytes) 2024-09-16T13:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout:1463 writesame 0xe1ce362 thru 0xe1da9fb data_size 0xb(0xc69a bytes) 2024-09-16T13:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout:1464 read 0x8fa0f85 thru 0x8fa7948 (0x69c4 bytes) 2024-09-16T13:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout:1465 trunc from 0xe1da9fc to 0x9e38161 2024-09-16T13:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout:1466 read 0x2d68b8c thru 0x2d6c27f (0x36f4 bytes) 2024-09-16T13:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout:1468 read 0x32352c8 thru 0x323b4a6 (0x61df bytes) 2024-09-16T13:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout:1470 read 0x46383f0 thru 0x463b271 (0x2e82 bytes) 2024-09-16T13:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout:1472 writesame 0xc04a564 thru 0xc04eb17 data_size 0x6(0x45b4 bytes) 2024-09-16T13:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout:1473 punch from 0xa632ce to 0xa6fc31, (0xc963 bytes) 2024-09-16T13:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout:1475 write 0x30e8a14 thru 0x30f75b5 (0xeba2 bytes) 2024-09-16T13:28:48.925 INFO:teuthology.orchestra.run.smithi076.stdout:1476 write 0xb30bf69 thru 0xb315135 (0x91cd bytes) 2024-09-16T13:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout:1477 trunc from 0xc04eb18 to 0x35e177b 2024-09-16T13:28:48.951 INFO:teuthology.orchestra.run.smithi076.stdout:1478 read 0x12dbe49 thru 0x12e35d2 (0x778a bytes) 2024-09-16T13:28:48.951 INFO:teuthology.orchestra.run.smithi076.stdout:1479 read 0x208b2ac thru 0x2095fbb (0xad10 bytes) 2024-09-16T13:28:48.963 INFO:teuthology.orchestra.run.smithi076.stdout:1480 write 0x7c509ad thru 0x7c5513f (0x4793 bytes) 2024-09-16T13:28:48.966 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:48.964+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:48.966 INFO:teuthology.orchestra.run.smithi076.stdout:1481 punch from 0x4144312 to 0x4149303, (0x4ff1 bytes) 2024-09-16T13:28:48.966 INFO:teuthology.orchestra.run.smithi076.stdout:1482 writesame 0x468c9c5 thru 0x4695493 data_size 0x67(0x8acf bytes) 2024-09-16T13:28:48.969 INFO:teuthology.orchestra.run.smithi076.stdout:1483 punch from 0x60fc6c4 to 0x610329e, (0x6bda bytes) 2024-09-16T13:28:48.981 INFO:teuthology.orchestra.run.smithi076.stdout:1485 write 0xdd499c6 thru 0xdd4f8dc (0x5f17 bytes) 2024-09-16T13:28:48.984 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:48.984 INFO:teuthology.orchestra.run.smithi076.stdout:1488 writesame 0x28e6303 thru 0x28eb775 data_size 0x1(0x5473 bytes) 2024-09-16T13:28:48.999 INFO:teuthology.orchestra.run.smithi076.stdout:1489 read 0x1485643 thru 0x1489a6b (0x4429 bytes) 2024-09-16T13:28:48.999 INFO:teuthology.orchestra.run.smithi076.stdout:1490 writesame 0xcfb8653 thru 0xcfbaa24 data_size 0x2(0x23d2 bytes) 2024-09-16T13:28:49.009 INFO:teuthology.orchestra.run.smithi076.stdout:1491 trunc from 0xdd4f8dd to 0x9e7ffda 2024-09-16T13:28:49.019 INFO:teuthology.orchestra.run.smithi076.stdout:1492 trunc from 0x9e7ffda to 0xabf9b7c 2024-09-16T13:28:49.024 INFO:teuthology.orchestra.run.smithi076.stdout:1495 punch from 0xa541fdb to 0xa547eb2, (0x5ed7 bytes) 2024-09-16T13:28:49.024 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:49.024 INFO:teuthology.orchestra.run.smithi076.stdout:1500 punch from 0x57d3583 to 0x57df70c, (0xc189 bytes) 2024-09-16T13:28:49.027 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.027 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.033 INFO:teuthology.orchestra.run.smithi076.stdout:1501 trunc from 0xabf9b7c to 0x281e4d8 2024-09-16T13:28:49.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:49.049 INFO:teuthology.orchestra.run.smithi076.stdout:1502 writesame 0x23f6482 thru 0x23f9d68 data_size 0x1(0x38e7 bytes) 2024-09-16T13:28:49.069 INFO:teuthology.orchestra.run.smithi076.stdout:1503 write 0x7dca79c thru 0x7dcbbe3 (0x1448 bytes) 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:1504 punch from 0x58f5492 to 0x5902af2, (0xd660 bytes) 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:1506 read 0x7bff8a4 thru 0x7c073a8 (0x7b05 bytes) 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:1509 read 0x529e865 thru 0x52a997f (0xb11b bytes) 2024-09-16T13:28:49.073 INFO:teuthology.orchestra.run.smithi076.stdout:1510 read 0x3e9e2c thru 0x3f5f9c (0xc171 bytes) 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:1511 read 0x98c25b thru 0x9907d9 (0x457f bytes) 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:1512 punch from 0x2a74a02 to 0x2a7d3e4, (0x89e2 bytes) 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:1514 read 0x3595f82 thru 0x35a571d (0xf79c bytes) 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:1515 punch from 0x2caefc6 to 0x2cb34e2, (0x451c bytes) 2024-09-16T13:28:49.074 INFO:teuthology.orchestra.run.smithi076.stdout:1516 write 0x4347ff2 thru 0x434f7d4 (0x77e3 bytes) 2024-09-16T13:28:49.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:28:49.077 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:49.075+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:49.078 INFO:teuthology.orchestra.run.smithi076.stdout:1517 punch from 0x63339b2 to 0x63378fc, (0x3f4a bytes) 2024-09-16T13:28:49.078 INFO:teuthology.orchestra.run.smithi076.stdout:1518 writesame 0x70ae345 thru 0x70ba08b data_size 0xb(0xbd47 bytes) 2024-09-16T13:28:49.084 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:49.084 INFO:teuthology.orchestra.run.smithi076.stdout:1520 punch from 0x26675e9 to 0x2673149, (0xbb60 bytes) 2024-09-16T13:28:49.084 INFO:teuthology.orchestra.run.smithi076.stdout:1522 write 0x12d6816 thru 0x12d82d4 (0x1abf bytes) 2024-09-16T13:28:49.089 INFO:teuthology.orchestra.run.smithi076.stdout:1523 trunc from 0x7dcbbe4 to 0x423e160 2024-09-16T13:28:49.100 INFO:teuthology.orchestra.run.smithi076.stdout:1526 read 0x2855ac3 thru 0x2864ed4 (0xf412 bytes) 2024-09-16T13:28:49.115 INFO:teuthology.orchestra.run.smithi076.stdout:1532 write 0xc69e9a2 thru 0xc6ac65f (0xdcbe bytes) 2024-09-16T13:28:49.118 INFO:teuthology.orchestra.run.smithi076.stdout:1534 write 0x16e9452 thru 0x16f1dc6 (0x8975 bytes) 2024-09-16T13:28:49.121 INFO:teuthology.orchestra.run.smithi076.stdout:1535 write 0x2f8e19b thru 0x2f9aa44 (0xc8aa bytes) 2024-09-16T13:28:49.124 INFO:teuthology.orchestra.run.smithi076.stdout:1536 clone 13 order 24 su 4194304 sc 2 2024-09-16T13:28:49.150 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:49.147+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:49.177 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.177 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:49.251 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:49.249+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:49.283 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.283 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:49.352 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:49.349+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:49.431 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.431 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:49.453 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:49.450+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:49.537 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.537 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:49.589 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:49.587+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:49.643 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.643 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:49.693 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:49.691+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:49.756 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:49.754+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:49.791 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.791 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:49.885 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:49.883+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:49.939 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:49.940 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:49.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:49.989 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:49.987+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:50.058 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:50.056+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:28:50.091 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:50.091 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:50.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:50.155 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone12 from 0xc6ac660 (overlap 0xd4259a) to 0x7f49c2 2024-09-16T13:28:50.159 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:50.156+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:50.240 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:50.240 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:50.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:50.266 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-09-16T13:28:50.266 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:50.263+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:50.305 INFO:teuthology.orchestra.run.smithi076.stdout:1538 writesame 0xe65fc88 thru 0xe662af5 data_size 0x15(0x2e6e bytes) 2024-09-16T13:28:50.305 INFO:teuthology.orchestra.run.smithi076.stdout:1541 write 0x546002d thru 0x546db06 (0xdada bytes) 2024-09-16T13:28:50.310 INFO:teuthology.orchestra.run.smithi076.stdout:1543 read 0xb4fdf8a thru 0xb4fe2b2 (0x329 bytes) 2024-09-16T13:28:50.310 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.321 INFO:teuthology.orchestra.run.smithi076.stdout:1546 trunc from 0xe662af6 to 0x4bf6877 2024-09-16T13:28:50.330 INFO:teuthology.orchestra.run.smithi076.stdout:1547 write 0x1cb11d7 thru 0x1cb208b (0xeb5 bytes) 2024-09-16T13:28:50.357 INFO:teuthology.orchestra.run.smithi076.stdout:1548 write 0xd15ae0a thru 0xd15e6f2 (0x38e9 bytes) 2024-09-16T13:28:50.360 INFO:teuthology.orchestra.run.smithi076.stdout:1549 write 0x2606165 thru 0x2606f68 (0xe04 bytes) 2024-09-16T13:28:50.360 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:50.358+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:50.366 INFO:teuthology.orchestra.run.smithi076.stdout:1551 writesame 0xb3449f1 thru 0xb352aa3 data_size 0x29(0xe0b3 bytes) 2024-09-16T13:28:50.375 INFO:teuthology.orchestra.run.smithi076.stdout:1552 writesame 0xde7cddb thru 0xde817e2 data_size 0x5c(0x4a08 bytes) 2024-09-16T13:28:50.375 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.375 INFO:teuthology.orchestra.run.smithi076.stdout:1554 punch from 0x9c96942 to 0x9ca0ac9, (0xa187 bytes) 2024-09-16T13:28:50.375 INFO:teuthology.orchestra.run.smithi076.stdout:1556 write 0x25a08d6 thru 0x25aa4ed (0x9c18 bytes) 2024-09-16T13:28:50.377 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:50.377 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:50.377 INFO:teuthology.orchestra.run.smithi076.stdout:1557 read 0xcee6046 thru 0xcee704a (0x1005 bytes) 2024-09-16T13:28:50.378 INFO:teuthology.orchestra.run.smithi076.stdout:1558 punch from 0x54f72ee to 0x5505d52, (0xea64 bytes) 2024-09-16T13:28:50.378 INFO:teuthology.orchestra.run.smithi076.stdout:1559 write 0x2a3516d thru 0x2a38571 (0x3405 bytes) 2024-09-16T13:28:50.383 INFO:teuthology.orchestra.run.smithi076.stdout:1560 read 0xd54099a thru 0xd54555e (0x4bc5 bytes) 2024-09-16T13:28:50.383 INFO:teuthology.orchestra.run.smithi076.stdout:1561 write 0x9c3786e thru 0x9c435c5 (0xbd58 bytes) 2024-09-16T13:28:50.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:50.388 INFO:teuthology.orchestra.run.smithi076.stdout:1562 read 0xb5cb8d4 thru 0xb5d00a2 (0x47cf bytes) 2024-09-16T13:28:50.388 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.397 INFO:teuthology.orchestra.run.smithi076.stdout:1564 trunc from 0xde817e3 to 0x5c51f96 2024-09-16T13:28:50.413 INFO:teuthology.orchestra.run.smithi076.stdout:1567 write 0x65628ff thru 0x656553e (0x2c40 bytes) 2024-09-16T13:28:50.416 INFO:teuthology.orchestra.run.smithi076.stdout:1568 read 0x5131d27 thru 0x5135a40 (0x3d1a bytes) 2024-09-16T13:28:50.426 INFO:teuthology.orchestra.run.smithi076.stdout:1569 write 0xa0fff9b thru 0xa103d82 (0x3de8 bytes) 2024-09-16T13:28:50.429 INFO:teuthology.orchestra.run.smithi076.stdout:1570 read 0x99d25fb thru 0x99d4b10 (0x2516 bytes) 2024-09-16T13:28:50.430 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.430 INFO:teuthology.orchestra.run.smithi076.stdout:1572 punch from 0x7417b0f to 0x742610a, (0xe5fb bytes) 2024-09-16T13:28:50.430 INFO:teuthology.orchestra.run.smithi076.stdout:1573 writesame 0x4b8877a thru 0x4b94b1d data_size 0x26(0xc3a4 bytes) 2024-09-16T13:28:50.437 INFO:teuthology.orchestra.run.smithi076.stdout:1574 trunc from 0xa103d83 to 0x5bf6c45 2024-09-16T13:28:50.461 INFO:teuthology.orchestra.run.smithi076.stdout:1575 write 0xb9e4953 thru 0xb9ed051 (0x86ff bytes) 2024-09-16T13:28:50.461 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:50.459+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:50.463 INFO:teuthology.orchestra.run.smithi076.stdout:1576 read 0x9dfa5ad thru 0x9e09952 (0xf3a6 bytes) 2024-09-16T13:28:50.464 INFO:teuthology.orchestra.run.smithi076.stdout:1577 read 0x6b4c352 thru 0x6b54efe (0x8bad bytes) 2024-09-16T13:28:50.464 INFO:teuthology.orchestra.run.smithi076.stdout:1579 punch from 0x2de2e09 to 0x2de96fe, (0x68f5 bytes) 2024-09-16T13:28:50.471 INFO:teuthology.orchestra.run.smithi076.stdout:1581 writesame 0xd9b4a50 thru 0xd9c207c data_size 0x1(0xd62d bytes) 2024-09-16T13:28:50.471 INFO:teuthology.orchestra.run.smithi076.stdout:1582 punch from 0x394d781 to 0x3955cf2, (0x8571 bytes) 2024-09-16T13:28:50.472 INFO:teuthology.orchestra.run.smithi076.stdout:1583 writesame 0x28f8a7f thru 0x2904589 data_size 0xb(0xbb0b bytes) 2024-09-16T13:28:50.479 INFO:teuthology.orchestra.run.smithi076.stdout:1584 writesame 0x5dd2f1f thru 0x5dd3d81 data_size 0x1(0xe63 bytes) 2024-09-16T13:28:50.482 INFO:teuthology.orchestra.run.smithi076.stdout:1585 read 0xc703b2d thru 0xc7086bd (0x4b91 bytes) 2024-09-16T13:28:50.482 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.482 INFO:teuthology.orchestra.run.smithi076.stdout:1588 punch from 0xa872d3a to 0xa87f704, (0xc9ca bytes) 2024-09-16T13:28:50.483 INFO:teuthology.orchestra.run.smithi076.stdout:1589 punch from 0xa3b934e to 0xa3c4736, (0xb3e8 bytes) 2024-09-16T13:28:50.483 INFO:teuthology.orchestra.run.smithi076.stdout:1591 write 0xcb563f2 thru 0xcb5ebd7 (0x87e6 bytes) 2024-09-16T13:28:50.486 INFO:teuthology.orchestra.run.smithi076.stdout:1593 read 0xb2737da thru 0xb2764d2 (0x2cf9 bytes) 2024-09-16T13:28:50.487 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.494 INFO:teuthology.orchestra.run.smithi076.stdout:1595 trunc from 0xd9c207d to 0x7af27e6 2024-09-16T13:28:50.498 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:50.498 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:50.505 INFO:teuthology.orchestra.run.smithi076.stdout:1596 trunc from 0x7af27e6 to 0x659b553 2024-09-16T13:28:50.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:50.510 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.510 INFO:teuthology.orchestra.run.smithi076.stdout:1598 punch from 0x419bef0 to 0x419d1f3, (0x1303 bytes) 2024-09-16T13:28:50.510 INFO:teuthology.orchestra.run.smithi076.stdout:1599 write 0x992c8f thru 0x992e4a (0x1bc bytes) 2024-09-16T13:28:50.522 INFO:teuthology.orchestra.run.smithi076.stdout:1600 read 0x4f746ad thru 0x4f80540 (0xbe94 bytes) 2024-09-16T13:28:50.522 INFO:teuthology.orchestra.run.smithi076.stdout:1601 read 0x24fd25c thru 0x2502895 (0x563a bytes) 2024-09-16T13:28:50.525 INFO:teuthology.orchestra.run.smithi076.stdout:1604 trunc from 0x659b553 to 0x41a5be2 2024-09-16T13:28:50.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:50.534 INFO:teuthology.orchestra.run.smithi076.stdout:1605 read 0x5c3adb thru 0x5c495a (0xe80 bytes) 2024-09-16T13:28:50.535 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.535 INFO:teuthology.orchestra.run.smithi076.stdout:1607 punch from 0xdb3a09 to 0xdb7e8f, (0x4486 bytes) 2024-09-16T13:28:50.549 INFO:teuthology.orchestra.run.smithi076.stdout:1608 write 0xc0b0215 thru 0xc0b8ef1 (0x8cdd bytes) 2024-09-16T13:28:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:1609 write 0x7b6908 thru 0x7c3586 (0xcc7f bytes) 2024-09-16T13:28:50.556 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.562 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:50.560+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:50.562 INFO:teuthology.orchestra.run.smithi076.stdout:1612 writesame 0xcb1156f thru 0xcb1a41c data_size 0x7(0x8eae bytes) 2024-09-16T13:28:50.568 INFO:teuthology.orchestra.run.smithi076.stdout:1614 trunc from 0xcb1a41d to 0x83e47a8 2024-09-16T13:28:50.577 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.581 INFO:teuthology.orchestra.run.smithi076.stdout:1616 writesame 0x856d900 thru 0x8571c9a data_size 0x9(0x439b bytes) 2024-09-16T13:28:50.581 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.581 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.581 INFO:teuthology.orchestra.run.smithi076.stdout:1620 write 0x6176251 thru 0x617b7e7 (0x5597 bytes) 2024-09-16T13:28:50.584 INFO:teuthology.orchestra.run.smithi076.stdout:1621 read 0x5909c96 thru 0x5910cfa (0x7065 bytes) 2024-09-16T13:28:50.585 INFO:teuthology.orchestra.run.smithi076.stdout:1622 read 0x732929b thru 0x7338e9d (0xfc03 bytes) 2024-09-16T13:28:50.585 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.585 INFO:teuthology.orchestra.run.smithi076.stdout:1624 write 0x69a415c thru 0x69ab41f (0x72c4 bytes) 2024-09-16T13:28:50.595 INFO:teuthology.orchestra.run.smithi076.stdout:1625 writesame 0x980e18b thru 0x9818e6c data_size 0x1(0xace2 bytes) 2024-09-16T13:28:50.595 INFO:teuthology.orchestra.run.smithi076.stdout:1626 writesame 0x1f33643 thru 0x1f41b00 data_size 0x43(0xe4be bytes) 2024-09-16T13:28:50.602 INFO:teuthology.orchestra.run.smithi076.stdout:1629 trunc from 0x9818e6d to 0x5b8c66d 2024-09-16T13:28:50.613 INFO:teuthology.orchestra.run.smithi076.stdout:1630 read 0x1ec1d95 thru 0x1ec8daf (0x701b bytes) 2024-09-16T13:28:50.613 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.627 INFO:teuthology.orchestra.run.smithi076.stdout:1633 write 0xd832b9c thru 0xd834df8 (0x225d bytes) 2024-09-16T13:28:50.628 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:50.628 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:50.630 INFO:teuthology.orchestra.run.smithi076.stdout:1634 writesame 0x27fd577 thru 0x27ff03d data_size 0x5(0x1ac7 bytes) 2024-09-16T13:28:50.632 INFO:teuthology.orchestra.run.smithi076.stdout:1635 write 0xa6a9806 thru 0xa6b473d (0xaf38 bytes) 2024-09-16T13:28:50.637 INFO:teuthology.orchestra.run.smithi076.stdout:1636 read 0xa1ad429 thru 0xa1b430a (0x6ee2 bytes) 2024-09-16T13:28:50.637 INFO:teuthology.orchestra.run.smithi076.stdout:1637 flatten 2024-09-16T13:28:50.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:50.643 INFO:teuthology.orchestra.run.smithi076.stdout:1638 write 0xa9e6ba4 thru 0xa9ee028 (0x7485 bytes) 2024-09-16T13:28:50.647 INFO:teuthology.orchestra.run.smithi076.stdout:1639 write 0x9960218 thru 0x99667fb (0x65e4 bytes) 2024-09-16T13:28:50.662 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:50.661+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:50.666 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:50.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:50.666 INFO:teuthology.orchestra.run.smithi076.stdout:1640 trunc from 0xd834df9 to 0xe582a1 2024-09-16T13:28:50.686 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:50.686 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:50.698 INFO:teuthology.orchestra.run.smithi076.stdout:1642 write 0xbdaebf7 thru 0xbdb1605 (0x2a0f bytes) 2024-09-16T13:28:50.701 INFO:teuthology.orchestra.run.smithi076.stdout:1644 writesame 0x2a21dfe thru 0x2a2a9fc data_size 0x1(0x8bff bytes) 2024-09-16T13:28:50.702 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:50.702 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:50.717 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:50.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:50.748 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:50.757 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:50.758 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:50.761 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.761 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.761 INFO:teuthology.orchestra.run.smithi076.stdout:1651 punch from 0xb14d1e9 to 0xb15724b, (0xa062 bytes) 2024-09-16T13:28:50.762 INFO:teuthology.orchestra.run.smithi076.stdout:1652 read 0x59a64be thru 0x59af7a1 (0x92e4 bytes) 2024-09-16T13:28:50.762 INFO:teuthology.orchestra.run.smithi076.stdout:1653 punch from 0x6382a36 to 0x6383c3b, (0x1205 bytes) 2024-09-16T13:28:50.763 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:50.761+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:50.764 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:50.764 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:50.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:50.771 INFO:teuthology.orchestra.run.smithi076.stdout:1654 trunc from 0xbdb1606 to 0x3b41013 2024-09-16T13:28:50.782 INFO:teuthology.orchestra.run.smithi076.stdout:1655 read 0xa15a99 thru 0xa24726 (0xec8e bytes) 2024-09-16T13:28:50.782 INFO:teuthology.orchestra.run.smithi076.stdout:1656 read 0xb95796 thru 0xb9dab2 (0x831d bytes) 2024-09-16T13:28:50.782 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:50.783 INFO:teuthology.orchestra.run.smithi076.stdout:1657 writesame 0x3865f6b thru 0x387392e data_size 0x4d(0xd9c4 bytes) 2024-09-16T13:28:50.784 INFO:teuthology.orchestra.run.smithi076.stdout:1658 punch from 0xf2beb4 to 0xf36ab8, (0xac04 bytes) 2024-09-16T13:28:50.791 INFO:teuthology.orchestra.run.smithi076.stdout:1659 write 0x470ba07 thru 0x4717b0e (0xc108 bytes) 2024-09-16T13:28:50.794 INFO:teuthology.orchestra.run.smithi076.stdout:1660 read 0x1f32889 thru 0x1f3d591 (0xad09 bytes) 2024-09-16T13:28:50.804 INFO:teuthology.orchestra.run.smithi076.stdout:1661 write 0x8fb7c72 thru 0x8fbda57 (0x5de6 bytes) 2024-09-16T13:28:50.819 INFO:teuthology.orchestra.run.smithi076.stdout:1662 write 0xe31b5b6 thru 0xe31bf45 (0x990 bytes) 2024-09-16T13:28:50.822 INFO:teuthology.orchestra.run.smithi076.stdout:1664 punch from 0xc263369 to 0xc26d6bd, (0xa354 bytes) 2024-09-16T13:28:50.822 INFO:teuthology.orchestra.run.smithi076.stdout:1666 write 0x66d1263 thru 0x66dcc07 (0xb9a5 bytes) 2024-09-16T13:28:50.826 INFO:teuthology.orchestra.run.smithi076.stdout:1667 writesame 0xa6b0425 thru 0xa6baff4 data_size 0x8(0xabd0 bytes) 2024-09-16T13:28:50.836 INFO:teuthology.orchestra.run.smithi076.stdout:1669 read 0x2035234 thru 0x203daa8 (0x8875 bytes) 2024-09-16T13:28:50.837 INFO:teuthology.orchestra.run.smithi076.stdout:1670 read 0x4b4fd88 thru 0x4b541a7 (0x4420 bytes) 2024-09-16T13:28:50.837 INFO:teuthology.orchestra.run.smithi076.stdout:1673 write 0x1ddc56a thru 0x1de6300 (0x9d97 bytes) 2024-09-16T13:28:50.840 INFO:teuthology.orchestra.run.smithi076.stdout:1674 write 0xba0dc93 thru 0xba16b71 (0x8edf bytes) 2024-09-16T13:28:50.844 INFO:teuthology.orchestra.run.smithi076.stdout:1676 writesame 0x3b15a86 thru 0x3b1bfde data_size 0x1(0x6559 bytes) 2024-09-16T13:28:50.863 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:50.861+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:50.864 INFO:teuthology.orchestra.run.smithi076.stdout:1677 punch from 0x75a67dd to 0x75af23a, (0x8a5d bytes) 2024-09-16T13:28:50.864 INFO:teuthology.orchestra.run.smithi076.stdout:1678 writesame 0x300502 thru 0x30f809 data_size 0x65(0xf308 bytes) 2024-09-16T13:28:50.865 INFO:teuthology.orchestra.run.smithi076.stdout:1679 punch from 0xa990283 to 0xa9951c7, (0x4f44 bytes) 2024-09-16T13:28:50.876 INFO:teuthology.orchestra.run.smithi076.stdout:1680 trunc from 0xe31bf46 to 0x539ab69 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.437661563999999, 2024-09-16T13:28:50.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:28:50.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:1146 4.f e37/32 ondisk, result = 0)", 2024-09-16T13:28:50.889 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.434695464000001, 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014045078000000001, 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445310+0000", 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445311+0000", 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6499999999999998e-07 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445315+0000", 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0319999999999997e-06 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445327+0000", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2495e-05 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.445333+0000", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9449999999999998e-06 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459267+0000", 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013933555 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459277+0000", 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0594000000000001e-05 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459355+0000", 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7591999999999998e-05 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.265262388, 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3373 4.b e38/32 4:d151a9d0:::rbd_data.11a976a8123d.000000000000000a:head v 38'14, mlcod=38'11)", 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.173959085, 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014151797000000001, 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706046+0000", 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706047+0000", 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706592+0000", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054438299999999995 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706613+0000", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1118999999999999e-05 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706620+0000", 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706670+0000", 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280999999999998e-05 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.706697+0000", 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7452000000000001e-05 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720145+0000", 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013447506999999999 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.720198+0000", 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3232999999999997e-05 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:50.902 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.030190513999999, 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:50.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:28:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:50.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.177490391999999, 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:28:50.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:28:50.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.156426484000001, 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:28:50.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.852353446, 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:28:50.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:28:50.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.924 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.7477596240000004, 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21707 4.14 e46/32 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head v 46'74, mlcod=45'72)", 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.7430163910000003, 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015037457000000001, 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000003e-07 2024-09-16T13:28:50.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137531+0000", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054107700000000005 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137543+0000", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2156e-05 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137548+0000", 2024-09-16T13:28:50.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.463e-06 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137896+0000", 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034767700000000001 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137917+0000", 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1523999999999999e-05 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.151679+0000", 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013761997 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152026+0000", 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034681000000000002 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.741215983, 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:28:50.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23819 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.0643149220000003, 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018047839999999999, 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23819 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815691+0000", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.054e-06 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815696+0000", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1749999999999996e-06 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815708+0000", 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1446e-05 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815715+0000", 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824348+0000", 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086330599999999997 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824416+0000", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8213999999999994e-05 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824679+0000", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026342200000000003 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826830+0000", 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002150749 2024-09-16T13:28:50.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833601+0000", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067705100000000004 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833698+0000", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7188000000000005e-05 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833738+0000", 2024-09-16T13:28:50.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0173000000000003e-05 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23820 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.0642842679999998, 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018255289000000001, 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.944 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23820 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:50.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815725+0000", 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7840000000000001e-06 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815732+0000", 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9040000000000003e-06 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815737+0000", 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5549999999999996e-06 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825011+0000", 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092747530000000002 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825098+0000", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6287999999999998e-05 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825475+0000", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037692200000000002 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826843+0000", 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001368403 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833748+0000", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069050989999999996 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833884+0000", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013585 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833976+0000", 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2219999999999995e-05 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.949 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23821 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.0642628749999998, 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018354471000000001, 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23821 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.950 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815743+0000", 2024-09-16T13:28:50.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.46e-07 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815747+0000", 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889999999999999e-06 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815756+0000", 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-06 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815760+0000", 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825875+0000", 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010114911000000001 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825919+0000", 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3572000000000002e-05 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826099+0000", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018034099999999999 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827140+0000", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010402899999999999 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833988+0000", 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068483349999999997 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834065+0000", 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7305999999999995e-05 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834097+0000", 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1492000000000003e-05 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25655 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:50.955 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2783255669999996, 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014517317, 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25655 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601681+0000", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6169999999999999e-06 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601688+0000", 2024-09-16T13:28:50.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.294e-06 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601703+0000", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4124999999999999e-05 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601709+0000", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1090000000000003e-06 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601751+0000", 2024-09-16T13:28:50.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2293000000000003e-05 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601810+0000", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8786000000000003e-05 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.602757+0000", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094687500000000002 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616128+0000", 2024-09-16T13:28:50.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013370885000000001 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616135+0000", 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9300000000000003e-06 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616171+0000", 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5196e-05 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616197+0000", 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2494562240000002, 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.961 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:28:50.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:28:50.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:28:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:28:50.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:28:50.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.2405796459999996, 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:28:50.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:28:50.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.0805854359999998, 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:28:50.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:28:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.0805524069999999, 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:28:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.0691885130000003, 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:28:50.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:50.988 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:50.989 INFO:teuthology.orchestra.run.smithi076.stdout:1681 punch from 0x31ee846 to 0x31f05d2, (0x1d8c bytes) 2024-09-16T13:28:50.989 INFO:teuthology.orchestra.run.smithi076.stdout:1682 read 0x820952 thru 0x82a4e1 (0x9b90 bytes) 2024-09-16T13:28:50.990 INFO:teuthology.orchestra.run.smithi076.stdout:1683 trunc from 0x539ab69 to 0x4356082 2024-09-16T13:28:50.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:1686 write 0xb7c41d2 thru 0xb7cc787 (0x85b6 bytes) 2024-09-16T13:28:50.991 INFO:teuthology.orchestra.run.smithi076.stdout:1687 writesame 0x8965c85 thru 0x8970339 data_size 0x1(0xa6b5 bytes) 2024-09-16T13:28:50.992 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:50.986+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:50.992 INFO:teuthology.orchestra.run.smithi076.stdout:1688 punch from 0x9272562 to 0x927e567, (0xc005 bytes) 2024-09-16T13:28:50.993 INFO:teuthology.orchestra.run.smithi076.stdout:1691 read 0x83e8e0f thru 0x83e8e18 (0xa bytes) 2024-09-16T13:28:50.993 INFO:teuthology.orchestra.run.smithi076.stdout:1693 write 0xa1d0951 thru 0xa1da454 (0x9b04 bytes) 2024-09-16T13:28:50.997 INFO:teuthology.orchestra.run.smithi076.stdout:1694 read 0xdb9e1 thru 0xdc6f9 (0xd19 bytes) 2024-09-16T13:28:50.997 INFO:teuthology.orchestra.run.smithi076.stdout:1695 write 0xb3ed9c thru 0xb454e4 (0x6749 bytes) 2024-09-16T13:28:50.999 INFO:teuthology.orchestra.run.smithi076.stdout:1697 read 0xa9881cf thru 0xa98f72d (0x755f bytes) 2024-09-16T13:28:51.000 INFO:teuthology.orchestra.run.smithi076.stdout:1698 read 0x6eabc33 thru 0x6eb6223 (0xa5f1 bytes) 2024-09-16T13:28:51.007 INFO:teuthology.orchestra.run.smithi076.stdout:1699 write 0xd2df7db thru 0xd2e741a (0x7c40 bytes) 2024-09-16T13:28:51.010 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.017 INFO:teuthology.orchestra.run.smithi076.stdout:1705 trunc from 0xd2e741b to 0x7401c84 2024-09-16T13:28:51.027 INFO:teuthology.orchestra.run.smithi076.stdout:1706 punch from 0x3422bcb to 0x34242d9, (0x170e bytes) 2024-09-16T13:28:51.027 INFO:teuthology.orchestra.run.smithi076.stdout:1707 read 0x522260b thru 0x52317ed (0xf1e3 bytes) 2024-09-16T13:28:51.027 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.034 INFO:teuthology.orchestra.run.smithi076.stdout:1709 trunc from 0x7401c84 to 0xd2d0495 2024-09-16T13:28:51.039 INFO:teuthology.orchestra.run.smithi076.stdout:1710 write 0x200cb74 thru 0x2018140 (0xb5cd bytes) 2024-09-16T13:28:51.042 INFO:teuthology.orchestra.run.smithi076.stdout:1711 write 0x6d118f0 thru 0x6d1389b (0x1fac bytes) 2024-09-16T13:28:51.053 INFO:teuthology.orchestra.run.smithi076.stdout:1712 write 0xe9c849c thru 0xe9d5976 (0xd4db bytes) 2024-09-16T13:28:51.056 INFO:teuthology.orchestra.run.smithi076.stdout:1713 writesame 0xa74c1cf thru 0xa751520 data_size 0x5a(0x5352 bytes) 2024-09-16T13:28:51.058 INFO:teuthology.orchestra.run.smithi076.stdout:1719 read 0x6e6f769 thru 0x6e7a5ae (0xae46 bytes) 2024-09-16T13:28:51.059 INFO:teuthology.orchestra.run.smithi076.stdout:1720 punch from 0x83da81b to 0x83da8fb, (0xe0 bytes) 2024-09-16T13:28:51.059 INFO:teuthology.orchestra.run.smithi076.stdout:1723 read 0x749ccf5 thru 0x74acc52 (0xff5e bytes) 2024-09-16T13:28:51.076 INFO:teuthology.orchestra.run.smithi076.stdout:1724 trunc from 0xe9d5977 to 0x1612a4 2024-09-16T13:28:51.077 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:51.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:28:51.077 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:51.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-09-16T13:28:51.090 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:51.088+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:51.101 INFO:teuthology.orchestra.run.smithi076.stdout:1726 trunc from 0x1612a4 to 0x6b41e5b 2024-09-16T13:28:51.106 INFO:teuthology.orchestra.run.smithi076.stdout:1728 writesame 0x6b2b671 thru 0x6b3083f data_size 0x9(0x51cf bytes) 2024-09-16T13:28:51.113 INFO:teuthology.orchestra.run.smithi076.stdout:1729 trunc from 0x6b41e5b to 0x5654599 2024-09-16T13:28:51.114 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.114 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:51.114 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:51.114 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:51.122 INFO:teuthology.orchestra.run.smithi076.stdout:1731 write 0x46b8000 thru 0x46c6021 (0xe022 bytes) 2024-09-16T13:28:51.124 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:51.138 INFO:teuthology.orchestra.run.smithi076.stdout:1732 write 0x9bd5f20 thru 0x9be19d9 (0xbaba bytes) 2024-09-16T13:28:51.144 INFO:teuthology.orchestra.run.smithi076.stdout:1733 trunc from 0x9be19da to 0x6d6029a 2024-09-16T13:28:51.156 INFO:teuthology.orchestra.run.smithi076.stdout:1734 trunc from 0x6d6029a to 0x5026d44 2024-09-16T13:28:51.162 INFO:teuthology.orchestra.run.smithi076.stdout:1735 punch from 0xc92f19 to 0xc98f7f, (0x6066 bytes) 2024-09-16T13:28:51.164 INFO:teuthology.orchestra.run.smithi076.stdout:1736 trunc from 0x5026d44 to 0x64cd298 2024-09-16T13:28:51.175 INFO:teuthology.orchestra.run.smithi076.stdout:1737 write 0x75b9716 thru 0x75bc55f (0x2e4a bytes) 2024-09-16T13:28:51.179 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.179 INFO:teuthology.orchestra.run.smithi076.stdout:1741 write 0x218adc4 thru 0x218bbef (0xe2c bytes) 2024-09-16T13:28:51.182 INFO:teuthology.orchestra.run.smithi076.stdout:1743 read 0x5af5489 thru 0x5b02c09 (0xd781 bytes) 2024-09-16T13:28:51.182 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.190 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:51.189+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:51.194 INFO:teuthology.orchestra.run.smithi076.stdout:1745 writesame 0xd35f405 thru 0xd36b803 data_size 0x4b(0xc3ff bytes) 2024-09-16T13:28:51.194 INFO:teuthology.orchestra.run.smithi076.stdout:1746 read 0x48aef19 thru 0x48b2f2a (0x4012 bytes) 2024-09-16T13:28:51.194 INFO:teuthology.orchestra.run.smithi076.stdout:1747 write 0x8d4c08c thru 0x8d593f8 (0xd36d bytes) 2024-09-16T13:28:51.210 INFO:teuthology.orchestra.run.smithi076.stdout:1748 trunc from 0xd36b804 to 0x4e70914 2024-09-16T13:28:51.221 INFO:teuthology.orchestra.run.smithi076.stdout:1749 read 0x1fc878f thru 0x1fd4226 (0xba98 bytes) 2024-09-16T13:28:51.229 INFO:teuthology.orchestra.run.smithi076.stdout:1750 trunc from 0x4e70914 to 0xbf09b93 2024-09-16T13:28:51.236 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.236 INFO:teuthology.orchestra.run.smithi076.stdout:1752 write 0x6c47ca1 thru 0x6c5100a (0x936a bytes) 2024-09-16T13:28:51.240 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.240 INFO:teuthology.orchestra.run.smithi076.stdout:1755 read 0x9148747 thru 0x914b628 (0x2ee2 bytes) 2024-09-16T13:28:51.240 INFO:teuthology.orchestra.run.smithi076.stdout:1756 read 0xbac0b97 thru 0xbacf508 (0xe972 bytes) 2024-09-16T13:28:51.240 INFO:teuthology.orchestra.run.smithi076.stdout:1757 write 0x25ee8a9 thru 0x25f4e9b (0x65f3 bytes) 2024-09-16T13:28:51.247 INFO:teuthology.orchestra.run.smithi076.stdout:1758 trunc from 0xbf09b93 to 0x8ad2378 2024-09-16T13:28:51.253 INFO:teuthology.orchestra.run.smithi076.stdout:1759 write 0x35ff313 thru 0x360ebe1 (0xf8cf bytes) 2024-09-16T13:28:51.255 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.255 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:51.255 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:51.255 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:51.255 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:51.256 INFO:teuthology.orchestra.run.smithi076.stdout:1761 punch from 0x4d7dbcc to 0x4d8874e, (0xab82 bytes) 2024-09-16T13:28:51.256 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.257 INFO:teuthology.orchestra.run.smithi076.stdout:1763 punch from 0x6be0323 to 0x6be2e2e, (0x2b0b bytes) 2024-09-16T13:28:51.257 INFO:teuthology.orchestra.run.smithi076.stdout:1764 trunc from 0x8ad2378 to 0x8cc1d86 2024-09-16T13:28:51.262 INFO:teuthology.orchestra.run.smithi076.stdout:1765 read 0x62ae877 thru 0x62be6b2 (0xfe3c bytes) 2024-09-16T13:28:51.262 INFO:teuthology.orchestra.run.smithi076.stdout:1766 write 0x53c0b3c thru 0x53c528c (0x4751 bytes) 2024-09-16T13:28:51.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:51.266 INFO:teuthology.orchestra.run.smithi076.stdout:1767 read 0x6fa9f13 thru 0x6fad2bc (0x33aa bytes) 2024-09-16T13:28:51.269 INFO:teuthology.orchestra.run.smithi076.stdout:1771 trunc from 0x8cc1d86 to 0xbd16852 2024-09-16T13:28:51.275 INFO:teuthology.orchestra.run.smithi076.stdout:1773 read 0x330cc31 thru 0x3318112 (0xb4e2 bytes) 2024-09-16T13:28:51.275 INFO:teuthology.orchestra.run.smithi076.stdout:1774 writesame 0x4b5f221 thru 0x4b6884d data_size 0x37(0x962d bytes) 2024-09-16T13:28:51.277 INFO:teuthology.orchestra.run.smithi076.stdout:1775 punch from 0x57c9173 to 0x57cfb19, (0x69a6 bytes) 2024-09-16T13:28:51.277 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.281 INFO:teuthology.orchestra.run.smithi076.stdout:1777 trunc from 0xbd16852 to 0x825c13c 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1778 read 0x3139385 thru 0x3148610 (0xf28c bytes) 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1779 punch from 0x3ffcafd to 0x4009adc, (0xcfdf bytes) 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1781 read 0x629d189 thru 0x62a324d (0x60c5 bytes) 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1782 punch from 0x6104468 to 0x6105756, (0x12ee bytes) 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1784 read 0x1b10eb2 thru 0x1b1811c (0x726b bytes) 2024-09-16T13:28:51.288 INFO:teuthology.orchestra.run.smithi076.stdout:1785 read 0x594f3c6 thru 0x5959c20 (0xa85b bytes) 2024-09-16T13:28:51.289 INFO:teuthology.orchestra.run.smithi076.stdout:1786 read 0x309c02d thru 0x30a0f56 (0x4f2a bytes) 2024-09-16T13:28:51.289 INFO:teuthology.orchestra.run.smithi076.stdout:1787 write 0x23ee9cd thru 0x23f6a2c (0x8060 bytes) 2024-09-16T13:28:51.291 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:51.289+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:51.291 INFO:teuthology.orchestra.run.smithi076.stdout:1789 punch from 0x76ae6ee to 0x76b33a8, (0x4cba bytes) 2024-09-16T13:28:51.292 INFO:teuthology.orchestra.run.smithi076.stdout:1790 trunc from 0x825c13c to 0x8d170bb 2024-09-16T13:28:51.297 INFO:teuthology.orchestra.run.smithi076.stdout:1791 read 0x57bcf2b thru 0x57c428e (0x7364 bytes) 2024-09-16T13:28:51.298 INFO:teuthology.orchestra.run.smithi076.stdout:1792 punch from 0x9926f7 to 0x99560d, (0x2f16 bytes) 2024-09-16T13:28:51.298 INFO:teuthology.orchestra.run.smithi076.stdout:1793 write 0x376b707 thru 0x3775d75 (0xa66f bytes) 2024-09-16T13:28:51.300 INFO:teuthology.orchestra.run.smithi076.stdout:1796 writesame 0x267607d thru 0x267be47 data_size 0x1(0x5dcb bytes) 2024-09-16T13:28:51.335 INFO:teuthology.orchestra.run.smithi076.stdout:1797 punch from 0x20e9ee3 to 0x20ef20b, (0x5328 bytes) 2024-09-16T13:28:51.335 INFO:teuthology.orchestra.run.smithi076.stdout:1798 punch from 0x4d3a39d to 0x4d47e8b, (0xdaee bytes) 2024-09-16T13:28:51.335 INFO:teuthology.orchestra.run.smithi076.stdout:1801 write 0x333e24 thru 0x3394cc (0x56a9 bytes) 2024-09-16T13:28:51.343 INFO:teuthology.orchestra.run.smithi076.stdout:1802 writesame 0x9c3e692 thru 0x9c4214d data_size 0x2a(0x3abc bytes) 2024-09-16T13:28:51.351 INFO:teuthology.orchestra.run.smithi076.stdout:1803 write 0xb70895c thru 0xb708f57 (0x5fc bytes) 2024-09-16T13:28:51.353 INFO:teuthology.orchestra.run.smithi076.stdout:1804 read 0x45b79d6 thru 0x45b954f (0x1b7a bytes) 2024-09-16T13:28:51.353 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.354 INFO:teuthology.orchestra.run.smithi076.stdout:1807 read 0x443a5df thru 0x444584c (0xb26e bytes) 2024-09-16T13:28:51.359 INFO:teuthology.orchestra.run.smithi076.stdout:1808 trunc from 0xb708f58 to 0x6c6526a 2024-09-16T13:28:51.370 INFO:teuthology.orchestra.run.smithi076.stdout:1809 read 0x206e9f3 thru 0x207c42e (0xda3c bytes) 2024-09-16T13:28:51.370 INFO:teuthology.orchestra.run.smithi076.stdout:1811 write 0x144256d thru 0x1451fdc (0xfa70 bytes) 2024-09-16T13:28:51.377 INFO:teuthology.orchestra.run.smithi076.stdout:1813 read 0x5a1f2fa thru 0x5a2ef0b (0xfc12 bytes) 2024-09-16T13:28:51.377 INFO:teuthology.orchestra.run.smithi076.stdout:1814 writesame 0x592fd05 thru 0x5939e5a data_size 0x6b(0xa156 bytes) 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:1858 4.d e38/32 4:b4b3857b:::rbd_data.11a962dad130.000000000000001d:head v 38'15, mlcod=37'13)", 2024-09-16T13:28:51.382 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.305506315999999, 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006645443, 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073917+0000", 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.073918+0000", 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.018e-06 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075152+0000", 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012342570000000001 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075168+0000", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5841999999999998e-05 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075174+0000", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3539999999999996e-06 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075208+0000", 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3887000000000003e-05 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.075221+0000", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2845e-05 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.080526+0000", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053053010000000001 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.080562+0000", 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5939000000000003e-05 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1932 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.302243959999998, 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079948060000000001, 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1932 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077179+0000", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077180+0000", 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2024-09-16T13:28:51.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077480+0000", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029970499999999999 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077495+0000", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5277999999999998e-05 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077501+0000", 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9070000000000004e-06 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077556+0000", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5189999999999998e-05 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077635+0000", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8985000000000004e-05 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077860+0000", 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000224887 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084483+0000", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066231550000000004 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085091+0000", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00060761599999999995 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085139+0000", 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.812e-05 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.085174+0000", 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4567000000000003e-05 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1933 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.301912162000001, 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088826889999999992, 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1933 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077514+0000", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077526+0000", 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1834000000000001e-05 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077529+0000", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.76e-06 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083235+0000", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057052270000000002 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083314+0000", 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9680999999999998e-05 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083493+0000", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017889199999999999 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084872+0000", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013790250000000001 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086316+0000", 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014437289999999999 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086358+0000", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1962999999999998e-05 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086393+0000", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5657000000000001e-05 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2074 4.d e38/32 4:b0be95a4:::rbd_data.11a962dad130.000000000000002a:head v 38'19, mlcod=38'17)", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:51.397 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.283234480000001, 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071711229999999997, 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096188+0000", 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096190+0000", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1170000000000001e-06 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096783+0000", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059312600000000003 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096792+0000", 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5689999999999994e-06 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096798+0000", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3170000000000001e-06 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096848+0000", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0818e-05 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.096874+0000", 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.103325+0000", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064508050000000004 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.103360+0000", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5051e-05 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2075 4.d e38/32 4:b0be95a4:::rbd_data.11a962dad130.000000000000002a:head v 38'21, mlcod=38'17)", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:51.401 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.280387975, 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051515320000000003, 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099035+0000", 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099036+0000", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1699999999999997e-07 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099043+0000", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7449999999999995e-06 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099053+0000", 2024-09-16T13:28:51.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.815e-06 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.099057+0000", 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8999999999999999e-06 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.101217+0000", 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021601839999999999 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.101240+0000", 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2770000000000001e-05 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.104159+0000", 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002919081 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.104186+0000", 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7319999999999999e-05 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2319 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.262700928000001, 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080751409999999992, 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2319 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.406 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116724+0000", 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1619999999999998e-06 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116732+0000", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.554e-06 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116751+0000", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9454999999999999e-05 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116758+0000", 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0650000000000001e-06 2024-09-16T13:28:51.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116832+0000", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4058000000000001e-05 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116965+0000", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013282999999999999 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124797+0000", 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078320170000000001 2024-09-16T13:28:51.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.247704558999999, 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:28:51.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.415 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 38/32 rep_tid 220 entries 38'14 (0'0) error 4:9f82cb54:::rbd_data.11a962dad130.0000000000000049:head by client.4521.0:2478 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 37'12)", 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.237695776999999, 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051641090000000001, 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:51.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141727+0000", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5699999999999999e-07 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141734+0000", 2024-09-16T13:28:51.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4529999999999999e-06 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141749+0000", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4749999999999999e-05 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141755+0000", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3330000000000002e-06 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141816+0000", 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1124999999999995e-05 2024-09-16T13:28:51.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146891+0000", 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050751909999999997 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2615 4.4 e38/32 4:236e8c22:::rbd_data.11a962dad130.0000000000000047:head v 38'17, mlcod=37'15)", 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.218315462, 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054677249999999997, 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161107+0000", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161108+0000", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161204+0000", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6316000000000004e-05 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161214+0000", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0176e-05 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161219+0000", 2024-09-16T13:28:51.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161363+0000", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014346300000000001 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.161381+0000", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7706e-05 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166538+0000", 2024-09-16T13:28:51.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051578750000000001 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.166575+0000", 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6823999999999998e-05 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2694 4.d e38/32 4:b2b91969:::rbd_data.11a962dad130.000000000000005c:head v 38'27, mlcod=38'24)", 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.211048963, 2024-09-16T13:28:51.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005599026, 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168374+0000", 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168375+0000", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4000000000000001e-07 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168385+0000", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0088000000000001e-05 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168400+0000", 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4866e-05 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168405+0000", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8159999999999999e-06 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168443+0000", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7724e-05 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.168461+0000", 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7989999999999999e-05 2024-09-16T13:28:51.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.173938+0000", 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054768960000000002 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.173973+0000", 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4906000000000001e-05 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.427 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2814 4.1c 4:3b21c817:::rbd_data.11a962dad130.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.186339188000002, 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052003220000000003, 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2814 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:51.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193084+0000", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193086+0000", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193364+0000", 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027807999999999999 2024-09-16T13:28:51.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193381+0000", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949999999999999e-05 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193389+0000", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193435+0000", 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5717000000000003e-05 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193562+0000", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012698900000000001 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193796+0000", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000233434 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198030+0000", 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042348960000000001 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198216+0000", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018563 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198249+0000", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3123000000000001e-05 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198284+0000", 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4919999999999998e-05 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2815 4.1c 4:3b21c817:::rbd_data.11a962dad130.0000000000000010:head [truncate 196608] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.186022764000001, 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055168279999999997, 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2815 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193400+0000", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193401+0000", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193403+0000", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7429999999999998e-06 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193410+0000", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3169999999999998e-06 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.193414+0000", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3540000000000002e-06 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197099+0000", 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036846460000000002 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197160+0000", 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0717000000000002e-05 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.197286+0000", 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126799 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198454+0000", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001167403 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198869+0000", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00041520000000000001 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198896+0000", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7316000000000001e-05 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.198917+0000", 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0772999999999999e-05 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:7508 4.19 e41/32 4:9af47f66:::rbd_data.11a91a08d9ca.0000000000000012:head v 41'26, mlcod=41'23)", 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.819334829000001, 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065137279999999999, 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560088+0000", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560089+0000", 2024-09-16T13:28:51.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0410000000000001e-06 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560512+0000", 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000422619 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560529+0000", 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7592999999999999e-05 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560536+0000", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560580+0000", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.560602+0000", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1933000000000002e-05 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.566558+0000", 2024-09-16T13:28:51.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0059559110000000004 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.566602+0000", 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3581999999999997e-05 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 46/32 rep_tid 455 entries 46'49 (0'0) error 4:9f62d19d:::rbd_data.11a98415fd69.0000000000000034:head by client.4521.0:21931 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 46'47)", 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2134695729999994, 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076044759999999998, 2024-09-16T13:28:51.442 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165953+0000", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165954+0000", 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8999999999999996e-07 2024-09-16T13:28:51.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165961+0000", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9079999999999998e-06 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165974+0000", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2874000000000001e-05 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.165982+0000", 2024-09-16T13:28:51.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166056+0000", 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4771000000000006e-05 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173558+0000", 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075015560000000004 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21932 4.19 e46/32 4:989741d8:::rbd_data.11a98415fd69.0000000000000030:head v 46'50, mlcod=46'47)", 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2130943470000002, 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072626100000000001, 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166329+0000", 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166335+0000", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6470000000000003e-06 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166348+0000", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599e-05 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166354+0000", 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1270000000000002e-06 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166402+0000", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8764999999999998e-05 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.166419+0000", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6237000000000001e-05 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173562+0000", 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007143445 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173591+0000", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9135000000000001e-05 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(4.19 epoch 46/32 rep_tid 458 entries 46'51 (0'0) error 4:9c2347db:::rbd_data.11a98415fd69.0000000000000037:head by client.4521.0:21939 0.000000 -2 [r=-2+0b,r=0+0b] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 46'47)", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2123352700000005, 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006512866, 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167088+0000", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6400000000000001e-07 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167097+0000", 2024-09-16T13:28:51.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2619999999999998e-06 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167115+0000", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7849000000000001e-05 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167123+0000", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0690000000000002e-06 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.167178+0000", 2024-09-16T13:28:51.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5825000000000002e-05 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.173601+0000", 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064220969999999999 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21984 4.19 e46/32 4:9f82c0fa:::rbd_data.11a98415fd69.0000000000000039:head v 46'55, mlcod=46'51)", 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2026401369999995, 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066286280000000001, 2024-09-16T13:28:51.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176783+0000", 2024-09-16T13:28:51.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3e-07 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176791+0000", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1210000000000001e-06 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176803+0000", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2571e-05 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176809+0000", 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:28:51.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176865+0000", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5828000000000001e-05 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.176881+0000", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6359e-05 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.183355+0000", 2024-09-16T13:28:51.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064736849999999999 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.183411+0000", 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6823999999999997e-05 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:22015 4.4 e46/32 4:2249ef24:::rbd_object_map.11a98415fd69.000000000000000c:head v 46'57, mlcod=46'55)", 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1998069210000004, 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078776269999999999, 2024-09-16T13:28:51.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179616+0000", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179617+0000", 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2500000000000004e-07 2024-09-16T13:28:51.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179624+0000", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.328e-06 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179636+0000", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1659e-05 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179642+0000", 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8529999999999997e-06 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179679+0000", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6977000000000002e-05 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.179699+0000", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0630999999999999e-05 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187421+0000", 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077212310000000003 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187494+0000", 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3022999999999995e-05 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:22062 4.1c 4:3e94e98f:::rbd_data.11a98415fd69.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.1974905620000005, 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056513229999999998, 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.461 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 22062 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181932+0000", 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181934+0000", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5549999999999999e-06 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181940+0000", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3230000000000003e-06 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181955+0000", 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4859e-05 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.181962+0000", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5259999999999999e-06 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182007+0000", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5351e-05 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182085+0000", 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7828000000000001e-05 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.182284+0000", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198844 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187425+0000", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051414800000000004 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187501+0000", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6264e-05 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187549+0000", 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7729000000000003e-05 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.187584+0000", 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4564000000000003e-05 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:23483 4.d e46/32 4:b36208b4:::rbd_data.11a9ff22d86c.0000000000000034:head v 46'50, mlcod=46'48)", 2024-09-16T13:28:51.466 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.5986834180000002, 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061942059999999998, 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780740+0000", 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.780741+0000", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1179999999999999e-06 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781581+0000", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084065599999999996 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781600+0000", 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8430000000000001e-05 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781607+0000", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1840000000000002e-06 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781654+0000", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6737999999999999e-05 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.781678+0000", 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.438e-05 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.786888+0000", 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005210279 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.786934+0000", 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5420999999999998e-05 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.470 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:51.471 INFO:teuthology.orchestra.run.smithi076.stdout:1815 write 0xe7f41f0 thru 0xe7f6ee8 (0x2cf9 bytes) 2024-09-16T13:28:51.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:51.472 INFO:teuthology.orchestra.run.smithi076.stdout:1817 writesame 0xd8959a9 thru 0xd89dfc6 data_size 0x1(0x861e bytes) 2024-09-16T13:28:51.472 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T13:28:51.398+0000 7fb7555f1640 -1 mon.a@0(leader).osd e51 definitely_dead 0 2024-09-16T13:28:51.473 INFO:teuthology.orchestra.run.smithi076.stdout:1818 punch from 0x20679ab to 0x2073661, (0xbcb6 bytes) 2024-09-16T13:28:51.473 INFO:teuthology.orchestra.run.smithi076.stdout:1819 punch from 0x26c87e7 to 0x26d2313, (0x9b2c bytes) 2024-09-16T13:28:51.473 INFO:teuthology.orchestra.run.smithi076.stdout:1820 trunc from 0xe7f6ee9 to 0x281a5e8 2024-09-16T13:28:51.473 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:51.469+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:51.486 INFO:teuthology.orchestra.run.smithi076.stdout:1821 write 0x5634c1b thru 0x564288c (0xdc72 bytes) 2024-09-16T13:28:51.489 INFO:teuthology.orchestra.run.smithi076.stdout:1822 write 0x55611f1 thru 0x556b0d0 (0x9ee0 bytes) 2024-09-16T13:28:51.492 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.499 INFO:teuthology.orchestra.run.smithi076.stdout:1824 write 0x69a5b1b thru 0x69a7403 (0x18e9 bytes) 2024-09-16T13:28:51.510 INFO:teuthology.orchestra.run.smithi076.stdout:1826 trunc from 0x69a7404 to 0xdfc821d 2024-09-16T13:28:51.532 INFO:teuthology.orchestra.run.smithi076.stdout:1829 trunc from 0xdfc821d to 0x100971 2024-09-16T13:28:51.561 INFO:teuthology.orchestra.run.smithi076.stdout:1833 write 0x9324972 thru 0x932f2d2 (0xa961 bytes) 2024-09-16T13:28:51.563 INFO:teuthology.orchestra.run.smithi076.stdout:1834 write 0x5d8a8e9 thru 0x5d95a68 (0xb180 bytes) 2024-09-16T13:28:51.568 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.568 INFO:teuthology.orchestra.run.smithi076.stdout:1838 writesame 0x5bd79d6 thru 0x5bdae5b data_size 0x51(0x3486 bytes) 2024-09-16T13:28:51.570 INFO:teuthology.orchestra.run.smithi076.stdout:1839 write 0x8c398ee thru 0x8c43fee (0xa701 bytes) 2024-09-16T13:28:51.572 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:51.571+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:51.580 INFO:teuthology.orchestra.run.smithi076.stdout:1840 trunc from 0x932f2d3 to 0xd86f926 2024-09-16T13:28:51.591 INFO:teuthology.orchestra.run.smithi076.stdout:1841 trunc from 0xd86f926 to 0x83b6b15 2024-09-16T13:28:51.601 INFO:teuthology.orchestra.run.smithi076.stdout:1842 write 0x3031034 thru 0x303a8c9 (0x9896 bytes) 2024-09-16T13:28:51.604 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.604 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:51.604 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:51.604 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:51.605 INFO:teuthology.orchestra.run.smithi076.stdout:1843 write 0x4bee1cd thru 0x4bf5ae1 (0x7915 bytes) 2024-09-16T13:28:51.607 INFO:teuthology.orchestra.run.smithi076.stdout:1844 writesame 0x5f714e8 thru 0x5f72bc2 data_size 0x1(0x16db bytes) 2024-09-16T13:28:51.610 INFO:teuthology.orchestra.run.smithi076.stdout:1845 read 0x75ec4fe thru 0x75f99c0 (0xd4c3 bytes) 2024-09-16T13:28:51.614 INFO:teuthology.orchestra.run.smithi076.stdout:1846 trunc from 0x83b6b15 to 0x4eac123 2024-09-16T13:28:51.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:51.623 INFO:teuthology.orchestra.run.smithi076.stdout:1848 punch from 0x363a266 to 0x363a653, (0x3ed bytes) 2024-09-16T13:28:51.624 INFO:teuthology.orchestra.run.smithi076.stdout:1849 write 0x28207eb thru 0x2823ae4 (0x32fa bytes) 2024-09-16T13:28:51.625 INFO:teuthology.orchestra.run.smithi076.stdout:1850 read 0x3edd0b9 thru 0x3ee037d (0x32c5 bytes) 2024-09-16T13:28:51.629 INFO:teuthology.orchestra.run.smithi076.stdout:1851 trunc from 0x4eac123 to 0x8dddb70 2024-09-16T13:28:51.642 INFO:teuthology.orchestra.run.smithi076.stdout:1855 write 0xb4a692b thru 0xb4b1ca4 (0xb37a bytes) 2024-09-16T13:28:51.645 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.645 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.646 INFO:teuthology.orchestra.run.smithi076.stdout:1860 trunc from 0xb4b1ca5 to 0xa1aca59 2024-09-16T13:28:51.656 INFO:teuthology.orchestra.run.smithi076.stdout:1862 punch from 0x4dfeade to 0x4e071ee, (0x8710 bytes) 2024-09-16T13:28:51.656 INFO:teuthology.orchestra.run.smithi076.stdout:1863 punch from 0x8cae6e5 to 0x8cbb438, (0xcd53 bytes) 2024-09-16T13:28:51.659 INFO:teuthology.orchestra.run.smithi076.stdout:1864 trunc from 0xa1aca59 to 0x6fe0b7e 2024-09-16T13:28:51.673 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:51.672+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:51.677 INFO:teuthology.orchestra.run.smithi076.stdout:1865 write 0xd6575e3 thru 0xd65d17c (0x5b9a bytes) 2024-09-16T13:28:51.680 INFO:teuthology.orchestra.run.smithi076.stdout:1868 punch from 0x553f652 to 0x5545f90, (0x693e bytes) 2024-09-16T13:28:51.680 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.680 INFO:teuthology.orchestra.run.smithi076.stdout:1871 writesame 0x710052b thru 0x710c1b2 data_size 0xc(0xbc88 bytes) 2024-09-16T13:28:51.685 INFO:teuthology.orchestra.run.smithi076.stdout:1872 read 0x58106de thru 0x5811dc3 (0x16e6 bytes) 2024-09-16T13:28:51.685 INFO:teuthology.orchestra.run.smithi076.stdout:1873 read 0xbf01cb2 thru 0xbf11b05 (0xfe54 bytes) 2024-09-16T13:28:51.685 INFO:teuthology.orchestra.run.smithi076.stdout:1876 read 0x8b38dc2 thru 0x8b3b30d (0x254c bytes) 2024-09-16T13:28:51.686 INFO:teuthology.orchestra.run.smithi076.stdout:1877 write 0x4bbf2b8 thru 0x4bc558b (0x62d4 bytes) 2024-09-16T13:28:51.687 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.703 INFO:teuthology.orchestra.run.smithi076.stdout:1880 trunc from 0xd65d17d to 0x187bc8e 2024-09-16T13:28:51.735 INFO:teuthology.orchestra.run.smithi076.stdout:1881 write 0xd610acd thru 0xd6128cf (0x1e03 bytes) 2024-09-16T13:28:51.753 INFO:teuthology.orchestra.run.smithi076.stdout:1882 trunc from 0xd6128d0 to 0x2bc66d0 2024-09-16T13:28:51.764 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:51.764 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:51.775 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:51.773+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:51.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:51.778 INFO:teuthology.orchestra.run.smithi076.stdout:1883 write 0xa3ff850 thru 0xa400ecc (0x167d bytes) 2024-09-16T13:28:51.780 INFO:teuthology.orchestra.run.smithi076.stdout:1884 read 0x4140a0c thru 0x414dce5 (0xd2da bytes) 2024-09-16T13:28:51.789 INFO:teuthology.orchestra.run.smithi076.stdout:1885 writesame 0xc9c9413 thru 0xc9d7f8a data_size 0x6e(0xeb78 bytes) 2024-09-16T13:28:51.789 INFO:teuthology.orchestra.run.smithi076.stdout:1886 write 0x594eda2 thru 0x5954f03 (0x6162 bytes) 2024-09-16T13:28:51.802 INFO:teuthology.orchestra.run.smithi076.stdout:1887 trunc from 0xc9d7f8b to 0x43dda37 2024-09-16T13:28:51.814 INFO:teuthology.orchestra.run.smithi076.stdout:1888 punch from 0xe85e08 to 0xe90c2e, (0xae26 bytes) 2024-09-16T13:28:51.814 INFO:teuthology.orchestra.run.smithi076.stdout:1889 read 0xd67746 thru 0xd7724f (0xfb0a bytes) 2024-09-16T13:28:51.816 INFO:teuthology.orchestra.run.smithi076.stdout:1891 trunc from 0x43dda37 to 0x2950495 2024-09-16T13:28:51.826 INFO:teuthology.orchestra.run.smithi076.stdout:1892 punch from 0x17c7c2a to 0x17ce2f8, (0x66ce bytes) 2024-09-16T13:28:51.839 INFO:teuthology.orchestra.run.smithi076.stdout:1893 trunc from 0x2950495 to 0xdf1eeaf 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:1894 punch from 0xacf7998 to 0xad0072f, (0x8d97 bytes) 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:1897 read 0x9fb4a26 thru 0x9fb5339 (0x914 bytes) 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:1899 read 0x69dc8d8 thru 0x69e327c (0x69a5 bytes) 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.845 INFO:teuthology.orchestra.run.smithi076.stdout:1903 punch from 0x72c8a6 to 0x733a3c, (0x7196 bytes) 2024-09-16T13:28:51.846 INFO:teuthology.orchestra.run.smithi076.stdout:1904 punch from 0x81d786f to 0x81dfdb9, (0x854a bytes) 2024-09-16T13:28:51.853 INFO:teuthology.orchestra.run.smithi076.stdout:1905 trunc from 0xdf1eeaf to 0x71b656a 2024-09-16T13:28:51.858 INFO:teuthology.orchestra.run.smithi076.stdout:1906 write 0x5158d01 thru 0x516671d (0xda1d bytes) 2024-09-16T13:28:51.863 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:51.865 INFO:teuthology.orchestra.run.smithi076.stdout:1908 trunc from 0x71b656a to 0x4f7f62b 2024-09-16T13:28:51.871 INFO:teuthology.orchestra.run.smithi076.stdout:1909 punch from 0x13aaa9e to 0x13ab4ea, (0xa4c bytes) 2024-09-16T13:28:51.871 INFO:teuthology.orchestra.run.smithi076.stdout:1910 read 0x2687866 thru 0x268d297 (0x5a32 bytes) 2024-09-16T13:28:51.871 INFO:teuthology.orchestra.run.smithi076.stdout:1911 punch from 0x141a05d to 0x141ce61, (0x2e04 bytes) 2024-09-16T13:28:51.875 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:51.874+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:51.887 INFO:teuthology.orchestra.run.smithi076.stdout:1914 writesame 0xd577b9f thru 0xd580343 data_size 0x19(0x87a5 bytes) 2024-09-16T13:28:51.887 INFO:teuthology.orchestra.run.smithi076.stdout:1917 write 0x26f3dd thru 0x2767ae (0x73d2 bytes) 2024-09-16T13:28:51.889 INFO:teuthology.orchestra.run.smithi076.stdout:1918 read 0x625b4ba thru 0x625b751 (0x298 bytes) 2024-09-16T13:28:51.889 INFO:teuthology.orchestra.run.smithi076.stdout:1920 clone 14 order 18 su 2048 sc 10 2024-09-16T13:28:51.897 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:51.897 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2256 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.786724104000001, 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016169003000000001, 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2256 2024-09-16T13:28:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8100000000000002e-07 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108176+0000", 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108184+0000", 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1239999999999999e-06 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108187+0000", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112601+0000", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044130940000000002 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112684+0000", 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3038e-05 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112976+0000", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000292665 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116526+0000", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035497549999999999 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124246+0000", 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00771975 2024-09-16T13:28:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124298+0000", 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2688000000000001e-05 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124341+0000", 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2218999999999997e-05 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2260 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.785524057, 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015260647, 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109373+0000", 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.11e-06 2024-09-16T13:28:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109378+0000", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.276e-06 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109392+0000", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3873000000000001e-05 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109399+0000", 2024-09-16T13:28:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3900000000000004e-06 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113426+0000", 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040269540000000001 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113497+0000", 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1154000000000003e-05 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114052+0000", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000554742 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118779+0000", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047264669999999998 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124446+0000", 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056670360000000003 2024-09-16T13:28:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124495+0000", 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8887999999999997e-05 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124632+0000", 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013775699999999999 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2262 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.784174736000001, 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014235015, 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110722+0000", 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-07 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110727+0000", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8899999999999998e-06 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110737+0000", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0311e-05 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110743+0000", 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6470000000000003e-06 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114432+0000", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036886760000000001 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114502+0000", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0246000000000005e-05 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114821+0000", 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031939499999999999 2024-09-16T13:28:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118971+0000", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041499559999999998 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124857+0000", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005885782 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124912+0000", 2024-09-16T13:28:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4950000000000001e-05 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124956+0000", 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4165000000000002e-05 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2264 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.782736762999999, 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012912022, 2024-09-16T13:28:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2264 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112160+0000", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9199999999999999e-07 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112164+0000", 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3789999999999999e-06 2024-09-16T13:28:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112175+0000", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.062e-05 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112182+0000", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2910000000000002e-06 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115155+0000", 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00297255 2024-09-16T13:28:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115230+0000", 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5164000000000001e-05 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125071+0000", 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098410259999999992 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2317 4.10 4:0d5a9ded:::rbd_data.11a962dad130.0000000000000035:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.778444425, 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013390631, 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2317 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116452+0000", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116455+0000", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6510000000000001e-06 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116462+0000", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9659999999999999e-06 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116466+0000", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121568+0000", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051019619999999998 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121646+0000", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8288000000000007e-05 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121822+0000", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017609200000000001 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125896+0000", 2024-09-16T13:28:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040733840000000002 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129420+0000", 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003524241 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129772+0000", 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035258700000000002 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129842+0000", 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9506999999999997e-05 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2396 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'5, mlcod=37'3)", 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.764827378, 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013627254, 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130069+0000", 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8999999999999995e-07 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130195+0000", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012528800000000001 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130204+0000", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9259999999999995e-06 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130211+0000", 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.024e-06 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130789+0000", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057756300000000001 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130806+0000", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7314999999999999e-05 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143643+0000", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012836172 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143696+0000", 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3075999999999998e-05 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2397 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'7, mlcod=37'3)", 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.762511501999999, 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01459504, 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132385+0000", 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6700000000000003e-07 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132395+0000", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9620000000000005e-06 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132405+0000", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8600000000000005e-06 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132410+0000", 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0579999999999997e-06 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138648+0000", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062382610000000002 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138673+0000", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779000000000001e-05 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146940+0000", 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082671900000000006 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146979+0000", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9162999999999998e-05 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2475 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.755217043000002, 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014446321999999999, 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2475 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139680+0000", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.19e-06 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139685+0000", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3689999999999998e-06 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139698+0000", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2376e-05 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139704+0000", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.105e-06 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140099+0000", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000395413 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140182+0000", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2818999999999994e-05 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140518+0000", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033604199999999999 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144678+0000", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004159991 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154051+0000", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093730879999999999 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154086+0000", 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4598000000000003e-05 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154125+0000", 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9331e-05 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2525 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.749954465999998, 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012849027000000001, 2024-09-16T13:28:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2525 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144942+0000", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1060000000000001e-06 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145045+0000", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103089 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145058+0000", 2024-09-16T13:28:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2242000000000001e-05 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145063+0000", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7760000000000003e-06 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150004+0000", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049408229999999996 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150079+0000", 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4247999999999998e-05 2024-09-16T13:28:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150307+0000", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022807200000000001 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157405+0000", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070986110000000003 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157707+0000", 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030204199999999998 2024-09-16T13:28:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157749+0000", 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1573999999999999e-05 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157790+0000", 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1443999999999998e-05 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2526 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.749825078000001, 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013452830000000001, 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2526 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145074+0000", 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:28:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145082+0000", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2009999999999999e-06 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145086+0000", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999996e-06 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154239+0000", 2024-09-16T13:28:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091526520000000007 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154329+0000", 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0061000000000004e-05 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154521+0000", 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019246399999999999 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158411+0000", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038892010000000001 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158452+0000", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1776000000000001e-05 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158487+0000", 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4725999999999999e-05 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158523+0000", 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6443000000000003e-05 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:3297 4.17 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2069521~10445] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.227638418000002, 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030440354999999999, 2024-09-16T13:28:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3297 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667258+0000", 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668628+0000", 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001369501 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668651+0000", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3017999999999999e-05 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668658+0000", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668704+0000", 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5479000000000003e-05 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668825+0000", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121791 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.669018+0000", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019308300000000001 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.685320+0000", 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016301690000000001 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697626+0000", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012306375 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697668+0000", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999997e-05 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697698+0000", 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0006999999999999e-05 2024-09-16T13:28:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6967 4.17 4:ef36fa71:::rbd_data.11a976a8123d.0000000000000005:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.512182457, 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012205384999999999, 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6967 2024-09-16T13:28:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382714+0000", 2024-09-16T13:28:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382716+0000", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0619999999999999e-06 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382721+0000", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6500000000000001e-06 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382725+0000", 2024-09-16T13:28:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0729999999999998e-06 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394737+0000", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012011867000000001 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394816+0000", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9169e-05 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394919+0000", 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102157 2024-09-16T13:28:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8556 4.0 4:017e3810:::rbd_data.11a91a08d9ca.00000000000000d0:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.954340814, 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03408597, 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8556 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940556+0000", 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.127e-06 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940560+0000", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940570+0000", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302000000000001e-05 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940574+0000", 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2024-09-16T13:28:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940608+0000", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3748000000000003e-05 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940660+0000", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.942061+0000", 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014009750000000001 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974517+0000", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032456609999999997 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974529+0000", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2235000000000001e-05 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974593+0000", 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974641+0000", 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7459000000000001e-05 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8556 4.0 e42/32 ondisk, result = 0)", 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.948216675999999, 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027841225000000001, 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946680+0000", 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:28:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946683+0000", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6009999999999999e-06 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946694+0000", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0949e-05 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946699+0000", 2024-09-16T13:28:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7740000000000004e-06 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974498+0000", 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027798386000000001 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974509+0000", 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1157e-05 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974520+0000", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1517e-05 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8592 4.19 4:999f307f:::rbd_data.11a91a08d9ca.000000000000003a:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.867483212, 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020334331000000001, 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8592 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027414+0000", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2389999999999999e-06 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027418+0000", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3669999999999999e-06 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027429+0000", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0861e-05 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027435+0000", 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6319999999999996e-06 2024-09-16T13:28:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027481+0000", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6856e-05 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027621+0000", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000139989 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029356+0000", 2024-09-16T13:28:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001734097 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047641+0000", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018285019999999999 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047652+0000", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.183e-05 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047707+0000", 2024-09-16T13:28:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4729999999999999e-05 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047747+0000", 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-05 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8592 4.19 e42/32 ondisk, result = 0)", 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.861320694, 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014068058, 2024-09-16T13:28:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033576+0000", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033579+0000", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.23e-06 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033590+0000", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033596+0000", 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6890000000000001e-06 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047623+0000", 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014026874 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047633+0000", 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007e-05 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047643+0000", 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302999999999999e-05 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21649 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.7648691979999995, 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020572536999999998, 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21649 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130028+0000", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3510000000000001e-06 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130517+0000", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048930600000000001 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130532+0000", 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4752e-05 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130538+0000", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1689999999999997e-06 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131149+0000", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061055899999999999 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131241+0000", 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1978000000000006e-05 2024-09-16T13:28:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131807+0000", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056649900000000004 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141368+0000", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095605849999999999 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150523+0000", 2024-09-16T13:28:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009155046 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150564+0000", 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649000000000001e-05 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150599+0000", 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4643e-05 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21651 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.7643175400000004, 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024369312000000001, 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21651 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130579+0000", 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9999999999999998e-07 2024-09-16T13:28:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130581+0000", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.537e-06 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130587+0000", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130591+0000", 2024-09-16T13:28:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138200+0000", 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007608325 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138265+0000", 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5007999999999998e-05 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138437+0000", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017266100000000001 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141654+0000", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032171859999999999 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154881+0000", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013226236000000001 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154915+0000", 2024-09-16T13:28:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4471999999999998e-05 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154947+0000", 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2372000000000001e-05 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21707 4.14 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.7613215750000002, 2024-09-16T13:28:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019033448000000001, 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21707 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133576+0000", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5960000000000001e-06 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134873+0000", 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012973640000000001 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134891+0000", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7550999999999998e-05 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134899+0000", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1070000000000005e-06 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134970+0000", 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0741000000000006e-05 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.135346+0000", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000376497 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136623+0000", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012766260000000001 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.143975+0000", 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073526049999999999 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152504+0000", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085292530000000005 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152549+0000", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4740000000000002e-05 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152608+0000", 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8368000000000001e-05 2024-09-16T13:28:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21708 4.14 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.7599734149999993, 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019703347, 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21708 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134923+0000", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.06e-06 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134927+0000", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134935+0000", 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1359999999999997e-06 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134940+0000", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5700000000000003e-06 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141963+0000", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070231240000000004 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.142034+0000", 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1198000000000005e-05 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.142205+0000", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017075 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.144291+0000", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020860090000000002 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154543+0000", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010252733999999999 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154588+0000", 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4569e-05 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154626+0000", 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7755e-05 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:52.001 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:52.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:52.003 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:52.001+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:52.079 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.7. 2024-09-16T13:28:52.091 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:28:52.103 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:52.102+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:52.105 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:52.106 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:52.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:52.204 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:52.202+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:52.252 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.252 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:52.252 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:52.252 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:52.252 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:52.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:52.346 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:52.344+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:52.406 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.406 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:52.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:52.406 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:52.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3132 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 4368762~22099] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.792473978, 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024228047999999999, 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.407 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3132 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608311+0000", 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4300000000000002e-07 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614081+0000", 2024-09-16T13:28:52.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057699559999999997 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614094+0000", 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2672e-05 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614100+0000", 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.071e-06 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614155+0000", 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614248+0000", 2024-09-16T13:28:52.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2958e-05 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614457+0000", 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020964000000000001 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.629775+0000", 2024-09-16T13:28:52.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015317084 2024-09-16T13:28:52.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632471+0000", 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026965330000000001 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632510+0000", 2024-09-16T13:28:52.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8831000000000001e-05 2024-09-16T13:28:52.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632538+0000", 2024-09-16T13:28:52.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8330000000000002e-05 2024-09-16T13:28:52.419 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.419 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.419 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6979 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017954743000001, 2024-09-16T13:28:52.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014097060999999999, 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6979 2024-09-16T13:28:52.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.422 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:52.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:52.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0900000000000001e-07 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:52.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382835+0000", 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6929999999999998e-06 2024-09-16T13:28:52.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382842+0000", 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3640000000000004e-06 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396711+0000", 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013869545 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396799+0000", 2024-09-16T13:28:52.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7874999999999995e-05 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396926+0000", 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012751700000000001 2024-09-16T13:28:52.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6981 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:52.430 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017924464, 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014344342, 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.431 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6981 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.34e-07 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382861+0000", 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.003e-06 2024-09-16T13:28:52.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382864+0000", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8430000000000001e-06 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382866+0000", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8360000000000001e-06 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396969+0000", 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01410347 2024-09-16T13:28:52.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397099+0000", 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013018700000000001 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397204+0000", 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104769 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6996 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017881055, 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014259829, 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6996 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382904+0000", 2024-09-16T13:28:52.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5400000000000002e-07 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382905+0000", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382911+0000", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382914+0000", 2024-09-16T13:28:52.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3270000000000001e-06 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396956+0000", 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014041394 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397050+0000", 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4258000000000005e-05 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397163+0000", 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113002 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6997 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017866268000001, 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014462567000000001, 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.440 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6997 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5100000000000001e-07 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382920+0000", 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6950000000000001e-06 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382925+0000", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6619999999999998e-06 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397188+0000", 2024-09-16T13:28:52.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014258694000000001 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397262+0000", 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4282000000000005e-05 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397380+0000", 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118411 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6998 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017851559, 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014636431, 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6998 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.445 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7199999999999999e-07 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382940+0000", 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0090000000000001e-06 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382944+0000", 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9670000000000003e-06 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382946+0000", 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0549999999999998e-06 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397421+0000", 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014474792 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397498+0000", 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6746e-05 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397569+0000", 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1390000000000006e-05 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7000 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2555904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.01782845, 2024-09-16T13:28:52.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014812446, 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7000 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:52.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.51e-07 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382958+0000", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.827e-06 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382964+0000", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9900000000000002e-06 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382967+0000", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397606+0000", 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014638979999999999 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397684+0000", 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929000000000006e-05 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397768+0000", 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4515e-05 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.453 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7008 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017815383, 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014735491, 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7008 2024-09-16T13:28:52.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3599999999999999e-07 2024-09-16T13:28:52.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382970+0000", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1370000000000001e-06 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382973+0000", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9979999999999999e-06 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382980+0000", 2024-09-16T13:28:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0230000000000004e-06 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397590+0000", 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014609347 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397630+0000", 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0184e-05 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397704+0000", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4466000000000002e-05 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7011 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017797010000001, 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014843745, 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.458 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7011 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382988+0000", 2024-09-16T13:28:52.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382990+0000", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9810000000000002e-06 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382996+0000", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3400000000000003e-06 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382999+0000", 2024-09-16T13:28:52.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4709999999999999e-06 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397717+0000", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014718072 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397757+0000", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9796999999999999e-05 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397831+0000", 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3656000000000001e-05 2024-09-16T13:28:52.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7023 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017753462, 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015041545999999999, 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7023 2024-09-16T13:28:52.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1400000000000001e-07 2024-09-16T13:28:52.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383032+0000", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9800000000000002e-07 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383035+0000", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.768e-06 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383038+0000", 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.971e-06 2024-09-16T13:28:52.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397863+0000", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014825155 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397954+0000", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0958999999999994e-05 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398072+0000", 2024-09-16T13:28:52.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011848100000000001 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7024 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.017743572000001, 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015178159, 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.466 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7024 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1899999999999998e-07 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383042+0000", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.57e-06 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383047+0000", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-06 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383050+0000", 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398103+0000", 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052963000000001 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398151+0000", 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8863999999999998e-05 2024-09-16T13:28:52.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398219+0000", 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7291000000000003e-05 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.690445350999999, 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:28:52.470 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:28:52.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:28:52.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.680906240000001, 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:28:52.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:28:52.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25661 4.d 4:b5278be0:::rbd_data.11a926bfcc53.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.7702050620000005, 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022042392000000001, 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25661 2024-09-16T13:28:52.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630580+0000", 2024-09-16T13:28:52.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.37e-06 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630587+0000", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.477e-06 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630597+0000", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0473e-05 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630619+0000", 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2050000000000001e-05 2024-09-16T13:28:52.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630649+0000", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9513999999999999e-05 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630704+0000", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4753999999999998e-05 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.631810+0000", 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011062489999999999 2024-09-16T13:28:52.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652526+0000", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020715588 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652559+0000", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3859999999999998e-05 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652594+0000", 2024-09-16T13:28:52.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4749000000000002e-05 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652621+0000", 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7308e-05 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25661 4.d e48/32 ondisk, result = 0)", 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.7656232579999998, 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017464550999999998, 2024-09-16T13:28:52.485 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:52.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635162+0000", 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3200000000000004e-07 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635165+0000", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5449999999999999e-06 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635178+0000", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2461e-05 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635182+0000", 2024-09-16T13:28:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.403e-06 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652548+0000", 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017365642000000001 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652555+0000", 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:28:52.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652625+0000", 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0648000000000005e-05 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.7144495050000002, 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:28:52.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:52.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:28:52.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:28:52.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:28:52.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:28:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.7053996570000001, 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:28:52.496 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:28:52.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:26040 4.9 4:95b88f14:::rbd_data.11a931a7800.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6046880~12093] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:52.500 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.1871323499999997, 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014734529999999999, 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 26040 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213653+0000", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8999999999999995e-07 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214180+0000", 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00052780299999999997 2024-09-16T13:28:52.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214192+0000", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1609e-05 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214198+0000", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6149999999999996e-06 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214315+0000", 2024-09-16T13:28:52.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000117442 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214398+0000", 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2669000000000004e-05 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214806+0000", 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040854599999999997 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.223014+0000", 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082080009999999995 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228337+0000", 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053223439999999997 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228366+0000", 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9547e-05 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228386+0000", 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0163999999999999e-05 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:52.506 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.004038054, 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:28:52.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:28:52.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:28:52.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:28:52.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.99339100499999999, 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:28:52.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:28:52.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:28:52.515 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.516 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:52.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:52.518 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:52.515+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:52.618 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:52.616+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:52.658 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.658 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:52.658 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:52.658 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:52.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:52.718 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:52.716+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:52.764 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.765 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:52.765 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:52.765 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:52.765 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:52.773 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:52.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:52.841 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:52.839+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:52.913 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:52.913 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.273988747000001, 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:28:52.916 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:28:52.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:28:52.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:52.922 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.183458342000002, 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:52.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:28:52.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:28:52.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.928 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3536 4.3 e38/32 4:c24635eb:::rbd_data.11a976a8123d.0000000000000000:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.174297643999999, 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018037721, 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739179+0000", 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.316e-06 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742916+0000", 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037375210000000002 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742931+0000", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5413999999999998e-05 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742941+0000", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4290000000000007e-06 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742984+0000", 2024-09-16T13:28:52.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3022000000000002e-05 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742999+0000", 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5279000000000001e-05 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757171+0000", 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014172034999999999 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757215+0000", 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3705000000000003e-05 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.156329172, 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.933 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:28:52.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.74056702, 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:28:52.937 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:28:52.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:28:52.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.941 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.670230738000001, 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:28:52.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6960 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530804694, 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018031049, 2024-09-16T13:28:52.947 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6960 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382671+0000", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382672+0000", 2024-09-16T13:28:52.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.601e-06 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382678+0000", 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382681+0000", 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400597+0000", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017916527000000002 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400640+0000", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2394000000000001e-05 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400701+0000", 2024-09-16T13:28:52.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1334000000000006e-05 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6962 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530790021, 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018139466, 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.952 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6962 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:52.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382687+0000", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382694+0000", 2024-09-16T13:28:52.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7800000000000003e-06 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382697+0000", 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400722+0000", 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018024563 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400763+0000", 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1003e-05 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400824+0000", 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1496999999999998e-05 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6968 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.956 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530774087999999, 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018252758000000001, 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6968 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382703+0000", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382712+0000", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9600000000000006e-06 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382715+0000", 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5640000000000001e-06 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400844+0000", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018129125999999999 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400893+0000", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8869000000000003e-05 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400954+0000", 2024-09-16T13:28:52.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.02e-05 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6970 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530754764999999, 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018372506, 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.961 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6970 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382722+0000", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382728+0000", 2024-09-16T13:28:52.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382751+0000", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3697e-05 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400983+0000", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018231872999999999 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401028+0000", 2024-09-16T13:28:52.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5148999999999998e-05 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401093+0000", 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4222999999999998e-05 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6984 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530620753000001, 2024-09-16T13:28:52.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018365686999999999, 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6984 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382855+0000", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.06e-07 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382858+0000", 2024-09-16T13:28:52.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382868+0000", 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9450000000000005e-06 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382877+0000", 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3829999999999998e-06 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401119+0000", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018241384999999999 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401160+0000", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1393999999999997e-05 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401220+0000", 2024-09-16T13:28:52.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9558999999999997e-05 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6986 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530591926, 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018458407, 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.970 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6986 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:52.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382886+0000", 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.159e-06 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382892+0000", 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3690000000000003e-06 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382896+0000", 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401242+0000", 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018346475000000001 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401282+0000", 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401341+0000", 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8906999999999997e-05 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.974 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6990 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530574876999999, 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018563608999999998, 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6990 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382902+0000", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382909+0000", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382913+0000", 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.552e-06 2024-09-16T13:28:52.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401365+0000", 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018452196000000001 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401405+0000", 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0525999999999998e-05 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401464+0000", 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8226000000000001e-05 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6991 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2555904~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530558026, 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018669403000000001, 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6991 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382919+0000", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.671e-06 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382926+0000", 2024-09-16T13:28:52.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7460000000000003e-06 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401486+0000", 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018556492000000001 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401525+0000", 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9338999999999997e-05 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401586+0000", 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1124000000000006e-05 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7002 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.983 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.530541747000001, 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018778276, 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7002 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382942+0000", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4149999999999999e-06 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382949+0000", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2840000000000001e-06 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382953+0000", 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401609+0000", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018656005999999999 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401650+0000", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1077e-05 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401711+0000", 2024-09-16T13:28:52.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1245000000000001e-05 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7003 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.53051778, 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018881326, 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7003 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382959+0000", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.61e-06 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382977+0000", 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7997999999999999e-05 2024-09-16T13:28:52.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382985+0000", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929999999999998e-06 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401733+0000", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018748602 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401775+0000", 2024-09-16T13:28:52.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1751000000000001e-05 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401838+0000", 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3188000000000006e-05 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7028 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.529578088999999, 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018078818999999999, 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7028 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383898+0000", 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0249999999999999e-06 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383902+0000", 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.357e-06 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383912+0000", 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0356999999999999e-05 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383918+0000", 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3290000000000001e-06 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401861+0000", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017943489999999999 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401916+0000", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4722999999999998e-05 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401976+0000", 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537999999999998e-05 2024-09-16T13:28:52.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.2271089919999998, 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:52.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:28:52.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:28:53.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:53.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:53.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.2170197250000001, 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:53.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:53.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 2.0702310910000001, 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:53.007 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:28:53.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:28:53.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:28:53.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:53.013 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:53.013 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:53.013 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:53.014 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:53.015 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:53.015 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:53.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:53.016 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:53.012+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:53.016 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:53.016 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:53.016 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:53.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:53.017 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:53.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:53.017 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:53.018 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:53.018 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:53.018 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:53.083 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:28:53.115 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:53.113+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:53.157 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.158 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:53.216 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:53.214+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:53.304 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.305 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:53.316 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:53.314+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:53.411 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.411 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.417 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:53.416+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:53.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:53.519 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:53.517+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:53.559 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.559 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:53.619 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:53.618+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:53.665 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.665 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:53.758 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:53.756+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:53.811 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.811 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:53.821 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:53.819+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:53.921 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:53.920+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:53.923 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:53.923 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:53.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:54.022 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:54.021+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:54.070 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.070 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:54.123 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:54.121+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:54.211 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.211 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:54.224 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:54.222+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:54.326 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:54.324+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:54.337 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.337 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:54.427 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:54.425+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:54.445 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.445 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:54.538 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:54.536+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:54.593 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.593 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:54.643 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:54.641+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:54.729 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:54.728+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:54.739 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.739 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:54.830 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:54.828+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:54.845 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.845 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:54.854 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:54.931 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:54.930+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:54.994 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:54.994 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:55.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:55.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:55.046 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:55.044+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:55.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:28:55.104 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:55.104 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:55.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:55.133 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:55.131+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:55.163 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:55.163 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:55.193 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:55.193 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:55.213 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:55.213 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:55.229 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:55.229 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:55.233 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:55.231+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:55.245 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:55.256 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:55.256 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:55.256 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:28:55.257 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:28:55.257 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:55.261 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:55.261 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:55.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:55.278 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:55.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:55.293 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:55.293 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:55.346 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:55.344+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:55.377 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2256 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.266605224999999, 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016169003000000001, 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2256 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108172+0000", 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8100000000000002e-07 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108176+0000", 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108184+0000", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1239999999999999e-06 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.108187+0000", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112601+0000", 2024-09-16T13:28:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044130940000000002 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112684+0000", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3038e-05 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112976+0000", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000292665 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116526+0000", 2024-09-16T13:28:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035497549999999999 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124246+0000", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00771975 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124298+0000", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2688000000000001e-05 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124341+0000", 2024-09-16T13:28:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2218999999999997e-05 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2260 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.265405178000002, 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015260647, 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109372+0000", 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109373+0000", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.11e-06 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109378+0000", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.276e-06 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109392+0000", 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3873000000000001e-05 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.109399+0000", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3900000000000004e-06 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113426+0000", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040269540000000001 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.113497+0000", 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1154000000000003e-05 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114052+0000", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000554742 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118779+0000", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047264669999999998 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124446+0000", 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056670360000000003 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124495+0000", 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8887999999999997e-05 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124632+0000", 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013775699999999999 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2262 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.264055856999999, 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014235015, 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110721+0000", 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110722+0000", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-07 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110727+0000", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8899999999999998e-06 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110737+0000", 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0311e-05 2024-09-16T13:28:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.110743+0000", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6470000000000003e-06 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114432+0000", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036886760000000001 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114502+0000", 2024-09-16T13:28:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0246000000000005e-05 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.114821+0000", 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031939499999999999 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.118971+0000", 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041499559999999998 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124857+0000", 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005885782 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124912+0000", 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4950000000000001e-05 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124956+0000", 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4165000000000002e-05 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2264 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.262617884000001, 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012912022, 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2264 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112159+0000", 2024-09-16T13:28:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112160+0000", 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9199999999999999e-07 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112164+0000", 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3789999999999999e-06 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112175+0000", 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.062e-05 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.112182+0000", 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2910000000000002e-06 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115155+0000", 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00297255 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.115230+0000", 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5164000000000001e-05 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125071+0000", 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098410259999999992 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2317 4.10 4:0d5a9ded:::rbd_data.11a962dad130.0000000000000035:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.258325546000002, 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013390631, 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2317 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116451+0000", 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116452+0000", 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116455+0000", 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6510000000000001e-06 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116462+0000", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9659999999999999e-06 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116466+0000", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121568+0000", 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051019619999999998 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121646+0000", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8288000000000007e-05 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.121822+0000", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017609200000000001 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.125896+0000", 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040733840000000002 2024-09-16T13:28:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129420+0000", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003524241 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129772+0000", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035258700000000002 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.129842+0000", 2024-09-16T13:28:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9506999999999997e-05 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2396 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'5, mlcod=37'3)", 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.244708499000001, 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013627254, 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130068+0000", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130069+0000", 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8999999999999995e-07 2024-09-16T13:28:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130195+0000", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012528800000000001 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130204+0000", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9259999999999995e-06 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130211+0000", 2024-09-16T13:28:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.024e-06 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130789+0000", 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057756300000000001 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.130806+0000", 2024-09-16T13:28:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7314999999999999e-05 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143643+0000", 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012836172 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.143696+0000", 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3075999999999998e-05 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:2397 4.16 e38/32 4:6e85f3e1:::rbd_data.11a962dad130.000000000000003f:head v 38'7, mlcod=37'3)", 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.242392623000001, 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01459504, 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132384+0000", 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132385+0000", 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6700000000000003e-07 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132395+0000", 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9620000000000005e-06 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132405+0000", 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8600000000000005e-06 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.132410+0000", 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0579999999999997e-06 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138648+0000", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062382610000000002 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.138673+0000", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779000000000001e-05 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146940+0000", 2024-09-16T13:28:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082671900000000006 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.146979+0000", 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9162999999999998e-05 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2475 4.14 4:2d1c0c38:::rbd_object_map.11a962dad130.0000000000000006:head [call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.235098164, 2024-09-16T13:28:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014446321999999999, 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2475 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139679+0000", 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139680+0000", 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.19e-06 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139685+0000", 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3689999999999998e-06 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139698+0000", 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2376e-05 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.139704+0000", 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.105e-06 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140099+0000", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000395413 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140182+0000", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2818999999999994e-05 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.140518+0000", 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033604199999999999 2024-09-16T13:28:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144678+0000", 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004159991 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154051+0000", 2024-09-16T13:28:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093730879999999999 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154086+0000", 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4598000000000003e-05 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154125+0000", 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9331e-05 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2525 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.229835587, 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012849027000000001, 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2525 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144941+0000", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.144942+0000", 2024-09-16T13:28:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1060000000000001e-06 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145045+0000", 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103089 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145058+0000", 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2242000000000001e-05 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145063+0000", 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7760000000000003e-06 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150004+0000", 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049408229999999996 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150079+0000", 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4247999999999998e-05 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.150307+0000", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022807200000000001 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157405+0000", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070986110000000003 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157707+0000", 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030204199999999998 2024-09-16T13:28:55.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157749+0000", 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1573999999999999e-05 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.157790+0000", 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1443999999999998e-05 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2526 4.14 4:2bd25d26:::rbd_data.11a962dad130.000000000000003e:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.229706198999999, 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013452830000000001, 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2526 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145071+0000", 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145074+0000", 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145082+0000", 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2009999999999999e-06 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.145086+0000", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999996e-06 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154239+0000", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091526520000000007 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154329+0000", 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0061000000000004e-05 2024-09-16T13:28:55.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.154521+0000", 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019246399999999999 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158411+0000", 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038892010000000001 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158452+0000", 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1776000000000001e-05 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158487+0000", 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4725999999999999e-05 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.158523+0000", 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6443000000000003e-05 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:3297 4.17 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2069521~10445] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 20.707519539, 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030440354999999999, 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3297 2024-09-16T13:28:55.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667258+0000", 2024-09-16T13:28:55.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668628+0000", 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001369501 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668651+0000", 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3017999999999999e-05 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668658+0000", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668704+0000", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5479000000000003e-05 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668825+0000", 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121791 2024-09-16T13:28:55.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.669018+0000", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019308300000000001 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.685320+0000", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016301690000000001 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697626+0000", 2024-09-16T13:28:55.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012306375 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697668+0000", 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999997e-05 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697698+0000", 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0006999999999999e-05 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.440 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6967 4.17 4:ef36fa71:::rbd_data.11a976a8123d.0000000000000005:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.992063578, 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012205384999999999, 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6967 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382713+0000", 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382714+0000", 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382716+0000", 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0619999999999999e-06 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382721+0000", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6500000000000001e-06 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.382725+0000", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0729999999999998e-06 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394737+0000", 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012011867000000001 2024-09-16T13:28:55.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394816+0000", 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9169e-05 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.394919+0000", 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102157 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8556 4.0 4:017e3810:::rbd_data.11a91a08d9ca.00000000000000d0:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.434221935, 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03408597, 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8556 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940556+0000", 2024-09-16T13:28:55.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.127e-06 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940560+0000", 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940570+0000", 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302000000000001e-05 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940574+0000", 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940608+0000", 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3748000000000003e-05 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940660+0000", 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.942061+0000", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014009750000000001 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974517+0000", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032456609999999997 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974529+0000", 2024-09-16T13:28:55.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2235000000000001e-05 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974593+0000", 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974641+0000", 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7459000000000001e-05 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8556 4.0 e42/32 ondisk, result = 0)", 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.428097797, 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027841225000000001, 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:55.452 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946680+0000", 2024-09-16T13:28:55.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946683+0000", 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6009999999999999e-06 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946694+0000", 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0949e-05 2024-09-16T13:28:55.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946699+0000", 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7740000000000004e-06 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974498+0000", 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027798386000000001 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974509+0000", 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1157e-05 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974520+0000", 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1517e-05 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8592 4.19 4:999f307f:::rbd_data.11a91a08d9ca.000000000000003a:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:55.456 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.347364333000002, 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020334331000000001, 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8592 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:55.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027413+0000", 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027414+0000", 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2389999999999999e-06 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027418+0000", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3669999999999999e-06 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027429+0000", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0861e-05 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027435+0000", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6319999999999996e-06 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027481+0000", 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6856e-05 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027621+0000", 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000139989 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029356+0000", 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001734097 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047641+0000", 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018285019999999999 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047652+0000", 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.183e-05 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047707+0000", 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4729999999999999e-05 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047747+0000", 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-05 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8592 4.19 e42/32 ondisk, result = 0)", 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.341201815000002, 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014068058, 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033575+0000", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033576+0000", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033579+0000", 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.23e-06 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033590+0000", 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.033596+0000", 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6890000000000001e-06 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047623+0000", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014026874 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047633+0000", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007e-05 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.047643+0000", 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302999999999999e-05 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21649 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.244750319, 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020572536999999998, 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21649 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130028+0000", 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3510000000000001e-06 2024-09-16T13:28:55.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130517+0000", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048930600000000001 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130532+0000", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4752e-05 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130538+0000", 2024-09-16T13:28:55.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1689999999999997e-06 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131149+0000", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061055899999999999 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131241+0000", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1978000000000006e-05 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131807+0000", 2024-09-16T13:28:55.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056649900000000004 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141368+0000", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095605849999999999 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150523+0000", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009155046 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150564+0000", 2024-09-16T13:28:55.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649000000000001e-05 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150599+0000", 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4643e-05 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21651 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.244198661, 2024-09-16T13:28:55.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024369312000000001, 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21651 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130579+0000", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9999999999999998e-07 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130581+0000", 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.537e-06 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130587+0000", 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130591+0000", 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138200+0000", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007608325 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138265+0000", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5007999999999998e-05 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138437+0000", 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017266100000000001 2024-09-16T13:28:55.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141654+0000", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032171859999999999 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154881+0000", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013226236000000001 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154915+0000", 2024-09-16T13:28:55.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4471999999999998e-05 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154947+0000", 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2372000000000001e-05 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21707 4.14 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.241202696, 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019033448000000001, 2024-09-16T13:28:55.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21707 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133574+0000", 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.133576+0000", 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5960000000000001e-06 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134873+0000", 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012973640000000001 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134891+0000", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7550999999999998e-05 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134899+0000", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1070000000000005e-06 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134970+0000", 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0741000000000006e-05 2024-09-16T13:28:55.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.135346+0000", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000376497 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136623+0000", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012766260000000001 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.143975+0000", 2024-09-16T13:28:55.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073526049999999999 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152504+0000", 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085292530000000005 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152549+0000", 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4740000000000002e-05 2024-09-16T13:28:55.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152608+0000", 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8368000000000001e-05 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21708 4.14 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.239854535999999, 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019703347, 2024-09-16T13:28:55.484 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21708 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134922+0000", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134923+0000", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.06e-06 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134927+0000", 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:28:55.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134935+0000", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1359999999999997e-06 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.134940+0000", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5700000000000003e-06 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141963+0000", 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070231240000000004 2024-09-16T13:28:55.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.142034+0000", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1198000000000005e-05 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.142205+0000", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017075 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.144291+0000", 2024-09-16T13:28:55.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020860090000000002 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154543+0000", 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010252733999999999 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154588+0000", 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4569e-05 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:28:55.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154626+0000", 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7755e-05 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:28:55.490 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:55.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:55.492 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:55.489+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:55.592 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:55.590+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:55.603 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:55.603 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:55.603 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:55.603 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:55.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:55.695 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:55.694+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:55.751 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:55.751 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:55.751 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:55.751 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:55.751 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:55.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:55.801 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:55.799+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:55.894 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:55.892+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:55.900 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:55.900 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:55.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:55.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:55.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3132 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 4368762~22099] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 21.289527674999999, 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024228047999999999, 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3132 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.901 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608311+0000", 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4300000000000002e-07 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614081+0000", 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057699559999999997 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614094+0000", 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2672e-05 2024-09-16T13:28:55.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614100+0000", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.071e-06 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614155+0000", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614248+0000", 2024-09-16T13:28:55.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2958e-05 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614457+0000", 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020964000000000001 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.629775+0000", 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015317084 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632471+0000", 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026965330000000001 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632510+0000", 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8831000000000001e-05 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632538+0000", 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8330000000000002e-05 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6979 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.515008439999999, 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014097060999999999, 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6979 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382829+0000", 2024-09-16T13:28:55.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0900000000000001e-07 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382830+0000", 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382835+0000", 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6929999999999998e-06 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382842+0000", 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3640000000000004e-06 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396711+0000", 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013869545 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396799+0000", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7874999999999995e-05 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396926+0000", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012751700000000001 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6981 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:55.911 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514978160999998, 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014344342, 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6981 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382860+0000", 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.34e-07 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382861+0000", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.003e-06 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382864+0000", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8430000000000001e-06 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382866+0000", 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8360000000000001e-06 2024-09-16T13:28:55.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396969+0000", 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01410347 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397099+0000", 2024-09-16T13:28:55.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013018700000000001 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397204+0000", 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104769 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.916 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6996 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514934751999998, 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014259829, 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6996 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382903+0000", 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382904+0000", 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5400000000000002e-07 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382905+0000", 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382911+0000", 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382914+0000", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3270000000000001e-06 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.396956+0000", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014041394 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397050+0000", 2024-09-16T13:28:55.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4258000000000005e-05 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397163+0000", 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113002 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6997 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514919965000001, 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014462567000000001, 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6997 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382918+0000", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5100000000000001e-07 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382920+0000", 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6950000000000001e-06 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382925+0000", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6619999999999998e-06 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397188+0000", 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014258694000000001 2024-09-16T13:28:55.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397262+0000", 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4282000000000005e-05 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397380+0000", 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118411 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6998 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514905255999999, 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014636431, 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6998 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7199999999999999e-07 2024-09-16T13:28:55.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382940+0000", 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0090000000000001e-06 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382944+0000", 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9670000000000003e-06 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382946+0000", 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0549999999999998e-06 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397421+0000", 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014474792 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397498+0000", 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6746e-05 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397569+0000", 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1390000000000006e-05 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7000 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:7 [sparse-read 2555904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514882147000002, 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014812446, 2024-09-16T13:28:55.930 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7000 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382956+0000", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.51e-07 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382958+0000", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.827e-06 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382964+0000", 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9900000000000002e-06 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382967+0000", 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397606+0000", 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014638979999999999 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397684+0000", 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929000000000006e-05 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397768+0000", 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4515e-05 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7008 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.934 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.51486908, 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014735491, 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7008 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382969+0000", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3599999999999999e-07 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382970+0000", 2024-09-16T13:28:55.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1370000000000001e-06 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382973+0000", 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9979999999999999e-06 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382980+0000", 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0230000000000004e-06 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397590+0000", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014609347 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397630+0000", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0184e-05 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397704+0000", 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4466000000000002e-05 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7011 4.1 4:805865a3:::rbd_data.11a976a8123d.0000000000000006:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514850707000001, 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014843745, 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7011 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382987+0000", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382988+0000", 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382990+0000", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9810000000000002e-06 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382996+0000", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3400000000000003e-06 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382999+0000", 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4709999999999999e-06 2024-09-16T13:28:55.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397717+0000", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014718072 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397757+0000", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9796999999999999e-05 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397831+0000", 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3656000000000001e-05 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.942 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7023 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514807159, 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015041545999999999, 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7023 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.943 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383031+0000", 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1400000000000001e-07 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383032+0000", 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9800000000000002e-07 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383035+0000", 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.768e-06 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383038+0000", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.971e-06 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397863+0000", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014825155 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.397954+0000", 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0958999999999994e-05 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398072+0000", 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011848100000000001 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7024 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.514797268999999, 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015178159, 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.947 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7024 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:55.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1899999999999998e-07 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383042+0000", 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.57e-06 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383047+0000", 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-06 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383050+0000", 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398103+0000", 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052963000000001 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398151+0000", 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8863999999999998e-05 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398219+0000", 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7291000000000003e-05 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.187499047999999, 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:28:55.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:28:55.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.957 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.177959937000001, 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:28:55.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25661 4.d 4:b5278be0:::rbd_data.11a926bfcc53.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2672587590000006, 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022042392000000001, 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.962 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25661 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630580+0000", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.37e-06 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630587+0000", 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.477e-06 2024-09-16T13:28:55.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630597+0000", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0473e-05 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630619+0000", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2050000000000001e-05 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630649+0000", 2024-09-16T13:28:55.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9513999999999999e-05 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630704+0000", 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4753999999999998e-05 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.631810+0000", 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011062489999999999 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652526+0000", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020715588 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652559+0000", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3859999999999998e-05 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652594+0000", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4749000000000002e-05 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652621+0000", 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7308e-05 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25661 4.d e48/32 ondisk, result = 0)", 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2626769549999999, 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017464550999999998, 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635162+0000", 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3200000000000004e-07 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635165+0000", 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5449999999999999e-06 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635178+0000", 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2461e-05 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635182+0000", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.403e-06 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652548+0000", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017365642000000001 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652555+0000", 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652625+0000", 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0648000000000005e-05 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2115032019999994, 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.972 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:55.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:28:55.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2024533539999993, 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.978 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:28:55.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:28:55.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:26040 4.9 4:95b88f14:::rbd_data.11a931a7800.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6046880~12093] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.6841860470000007, 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014734529999999999, 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.982 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 26040 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213652+0000", 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.213653+0000", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8999999999999995e-07 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214180+0000", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00052780299999999997 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214192+0000", 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1609e-05 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214198+0000", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6149999999999996e-06 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214315+0000", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000117442 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214398+0000", 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2669000000000004e-05 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.214806+0000", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040854599999999997 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.223014+0000", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082080009999999995 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228337+0000", 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0053223439999999997 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228366+0000", 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9547e-05 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:47.228386+0000", 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0163999999999999e-05 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.5010917509999997, 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4904447019999996, 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:55.993 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:28:55.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:28:55.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:55.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:55.997 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:55.999 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:28:55.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:56.000 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:55.997+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:56.036 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-09-16T13:28:56.048 INFO:teuthology.orchestra.run.smithi076.stdout:1921 read 0x28eddf6 thru 0x28f3598 (0x57a3 bytes) 2024-09-16T13:28:56.049 INFO:teuthology.orchestra.run.smithi076.stdout:1924 write 0xa51292a thru 0xa5212a1 (0xe978 bytes) 2024-09-16T13:28:56.068 INFO:teuthology.orchestra.run.smithi076.stdout:1926 read 0x85cb768 thru 0x85d73ac (0xbc45 bytes) 2024-09-16T13:28:56.069 INFO:teuthology.orchestra.run.smithi076.stdout:1927 write 0xbad69d8 thru 0xbadbbba (0x51e3 bytes) 2024-09-16T13:28:56.100 INFO:teuthology.orchestra.run.smithi076.stdout:1928 trunc from 0xd580344 to 0x1ab4f2b 2024-09-16T13:28:56.101 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:56.099+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:30085 4.15 4:af4a51a0:::rbd_object_map.11a954136fc9:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:56.145736+0000", 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.0013292219999999999, 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013482749999999999, 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:28:56.148 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 30085 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145736+0000", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145736+0000", 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145738+0000", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6929999999999999e-06 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145746+0000", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145761+0000", 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5750999999999999e-05 2024-09-16T13:28:56.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145767+0000", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9800000000000003e-06 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145808+0000", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0747000000000001e-05 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.145889+0000", 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0871999999999994e-05 2024-09-16T13:28:56.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.146121+0000", 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000231652 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.146925+0000", 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080418500000000001 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:30085 4.15 e56/55 ondisk, result = 0)", 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:56.147041+0000", 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 2.4484000000000001e-05, 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335000000000006e-05, 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147041+0000", 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147041+0000", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147041+0000", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9500000000000001e-07 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147046+0000", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3730000000000003e-06 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147057+0000", 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.081e-05 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147062+0000", 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5969999999999999e-06 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147090+0000", 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7898e-05 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:56.147095+0000", 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8300000000000003e-06 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 2 2024-09-16T13:28:56.156 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:56.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:56.201 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:56.200+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:56.256 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.256 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:56.256 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:56.256 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:56.256 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:56.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:56.277 INFO:teuthology.orchestra.run.smithi076.stdout:1929 punch from 0xc75b9a to 0xc7ef98, (0x93fe bytes) 2024-09-16T13:28:56.278 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:56.298 INFO:teuthology.orchestra.run.smithi076.stdout:1932 write 0xec921fb thru 0xec9ff27 (0xdd2d bytes) 2024-09-16T13:28:56.307 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:56.305+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:56.325 INFO:teuthology.orchestra.run.smithi076.stdout:1933 trunc from 0xec9ff28 to 0x1fb80bf 2024-09-16T13:28:56.353 INFO:teuthology.orchestra.run.smithi076.stdout:1934 write 0xaffa800 thru 0xb007b65 (0xd366 bytes) 2024-09-16T13:28:56.371 INFO:teuthology.orchestra.run.smithi076.stdout:1935 punch from 0xdce8d6 to 0xdcea6e, (0x198 bytes) 2024-09-16T13:28:56.371 INFO:teuthology.orchestra.run.smithi076.stdout:1936 punch from 0xa1bd15a to 0xa1cc985, (0xf82b bytes) 2024-09-16T13:28:56.371 INFO:teuthology.orchestra.run.smithi076.stdout:1937 writesame 0x74289ee thru 0x743462f data_size 0x1(0xbc42 bytes) 2024-09-16T13:28:56.385 INFO:teuthology.orchestra.run.smithi076.stdout:1938 read 0x24154bd thru 0x241da99 (0x85dd bytes) 2024-09-16T13:28:56.385 INFO:teuthology.orchestra.run.smithi076.stdout:1939 write 0x11f36e4 thru 0x11f8850 (0x516d bytes) 2024-09-16T13:28:56.403 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:56.401+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:56.413 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 21.771804878000001, 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:28:56.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:28:56.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 21.681274472999998, 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.420 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:28:56.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:28:56.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3536 4.3 e38/32 4:c24635eb:::rbd_data.11a976a8123d.0000000000000000:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 21.672113775, 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018037721, 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739179+0000", 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.316e-06 2024-09-16T13:28:56.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742916+0000", 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037375210000000002 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742931+0000", 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5413999999999998e-05 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742941+0000", 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4290000000000007e-06 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742984+0000", 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3022000000000002e-05 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742999+0000", 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5279000000000001e-05 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757171+0000", 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014172034999999999 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757215+0000", 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3705000000000003e-05 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 21.654145303, 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:28:56.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:28:56.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.238383151000001, 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:56.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:28:56.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:28:56.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.168046869000001, 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:28:56.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:28:56.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:28:56.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6960 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:56.444 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028620825000001, 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018031049, 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6960 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382671+0000", 2024-09-16T13:28:56.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382672+0000", 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.601e-06 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382678+0000", 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382681+0000", 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400597+0000", 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017916527000000002 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400640+0000", 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2394000000000001e-05 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400701+0000", 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1334000000000006e-05 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6962 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028606151999998, 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018139466, 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.449 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6962 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:56.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382687+0000", 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:28:56.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382694+0000", 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7800000000000003e-06 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382697+0000", 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400722+0000", 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018024563 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400763+0000", 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1003e-05 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400824+0000", 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1496999999999998e-05 2024-09-16T13:28:56.453 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6968 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028590219000002, 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018252758000000001, 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6968 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382703+0000", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382712+0000", 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9600000000000006e-06 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382715+0000", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5640000000000001e-06 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400844+0000", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018129125999999999 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400893+0000", 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8869000000000003e-05 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400954+0000", 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.02e-05 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6970 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028570896000002, 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018372506, 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.458 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6970 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382722+0000", 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382728+0000", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382751+0000", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3697e-05 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400983+0000", 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018231872999999999 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401028+0000", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5148999999999998e-05 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401093+0000", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4222999999999998e-05 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6984 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028436884000001, 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018365686999999999, 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6984 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382855+0000", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.06e-07 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382858+0000", 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382868+0000", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9450000000000005e-06 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382877+0000", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3829999999999998e-06 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401119+0000", 2024-09-16T13:28:56.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018241384999999999 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401160+0000", 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1393999999999997e-05 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401220+0000", 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9558999999999997e-05 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6986 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028408057, 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018458407, 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6986 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382886+0000", 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.159e-06 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382892+0000", 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3690000000000003e-06 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382896+0000", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401242+0000", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018346475000000001 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401282+0000", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401341+0000", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8906999999999997e-05 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6990 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028391008, 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018563608999999998, 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6990 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382902+0000", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382909+0000", 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382913+0000", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.552e-06 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401365+0000", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018452196000000001 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401405+0000", 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0525999999999998e-05 2024-09-16T13:28:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401464+0000", 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8226000000000001e-05 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6991 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2555904~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028374156999998, 2024-09-16T13:28:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018669403000000001, 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6991 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382919+0000", 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.671e-06 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382926+0000", 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7460000000000003e-06 2024-09-16T13:28:56.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401486+0000", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018556492000000001 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401525+0000", 2024-09-16T13:28:56.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9338999999999997e-05 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401586+0000", 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1124000000000006e-05 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7002 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.480 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028357878000001, 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018778276, 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7002 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:28:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382942+0000", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4149999999999999e-06 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382949+0000", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2840000000000001e-06 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382953+0000", 2024-09-16T13:28:56.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401609+0000", 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018656005999999999 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401650+0000", 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1077e-05 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401711+0000", 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1245000000000001e-05 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7003 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.028333911000001, 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018881326, 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.485 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7003 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382959+0000", 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.61e-06 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382977+0000", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7997999999999999e-05 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382985+0000", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929999999999998e-06 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401733+0000", 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018748602 2024-09-16T13:28:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401775+0000", 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1751000000000001e-05 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401838+0000", 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3188000000000006e-05 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7028 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.027394220000001, 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018078818999999999, 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7028 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:56.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383898+0000", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0249999999999999e-06 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383902+0000", 2024-09-16T13:28:56.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.357e-06 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383912+0000", 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0356999999999999e-05 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383918+0000", 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3290000000000001e-06 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401861+0000", 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017943489999999999 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401916+0000", 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4722999999999998e-05 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401976+0000", 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537999999999998e-05 2024-09-16T13:28:56.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7249251230000002, 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.494 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:28:56.499 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.7148358560000005, 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.503 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.5680472219999997, 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:28:56.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:28:56.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:28:56.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:28:56.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.510 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:56.512 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:56.511+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:56.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:28:56.559 INFO:teuthology.orchestra.run.smithi076.stdout:1940 write 0x9c97257 thru 0x9ca0089 (0x8e33 bytes) 2024-09-16T13:28:56.571 INFO:teuthology.orchestra.run.smithi076.stdout:1941 read 0x2c560ea thru 0x2c5cbe4 (0x6afb bytes) 2024-09-16T13:28:56.572 INFO:teuthology.orchestra.run.smithi076.stdout:1942 punch from 0xa98527 to 0xa9c23b, (0x3d14 bytes) 2024-09-16T13:28:56.574 INFO:teuthology.orchestra.run.smithi076.stdout:1943 trunc from 0xb007b66 to 0xd174b3c 2024-09-16T13:28:56.583 INFO:teuthology.orchestra.run.smithi076.stdout:1944 punch from 0x8c2db98 to 0x8c3ad57, (0xd1bf bytes) 2024-09-16T13:28:56.583 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:56.583 INFO:teuthology.orchestra.run.smithi076.stdout:1947 flatten 2024-09-16T13:28:56.613 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:56.611+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:56.656 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.656 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:56.657 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:56.657 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:56.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:28:56.750 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:56.748+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:56.806 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.806 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:56.806 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:56.806 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:56.806 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:56.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:28:56.816 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:56.814+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:56.917 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:56.915+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.266316849999999, 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:56.956 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:28:56.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.256745372000001, 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:28:56.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7494 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10944512~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.409356907999999, 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036610563999999998, 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7494 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:56.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543980+0000", 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.85e-07 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543983+0000", 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543985+0000", 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580303+0000", 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036318691 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580359+0000", 2024-09-16T13:28:56.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5072999999999999e-05 2024-09-16T13:28:56.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580590+0000", 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023103699999999999 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.973 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.409348959999999, 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:28:56.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:56.978 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.40933669, 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:28:56.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:28:56.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.409325845000001, 2024-09-16T13:28:56.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.409316431000001, 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:28:56.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8154 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663084971, 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036367568000000003, 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8154 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:56.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290252+0000", 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4799999999999997e-07 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290255+0000", 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9239999999999999e-06 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290257+0000", 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7859999999999999e-06 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326341+0000", 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036084789999999999 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326384+0000", 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509999999999998e-05 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326619+0000", 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023450899999999999 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:56.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663077298000001, 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:56.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:56.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:56.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:28:57.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663068953, 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:28:57.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:28:57.005 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663058259, 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.006 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663049270999998, 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.011 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:28:57.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:28:57.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.015 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663041530000001, 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:28:57.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:57.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:28:57.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663032971, 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:28:57.020 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:28:57.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:28:57.024 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.025 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663024094000001, 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:28:57.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:28:57.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663010229000001, 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.030 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:28:57.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.034 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.663001762, 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.035 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:57.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:57.039 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.662992959, 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:57.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:28:57.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:28:57.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.0395229009999998, 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:28:57.044 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:28:57.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:28:57.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:28:57.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:28:57.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.0262942199999996, 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:28:57.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:28:57.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.054 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:57.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:28:57.056 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:57.053+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:57.094 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:28:57.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T13:28:57.156 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:57.154+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:57.195 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:57.195 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:57.195 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:28:57.195 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:57.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:28:57.257 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:57.255+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:57.300 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:57.300 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:28:57.301 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:28:57.301 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:28:57.301 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:57.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:28:57.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:28:57.355 INFO:teuthology.orchestra.run.smithi076.stdout:1948 read 0x8661f1 thru 0x86872f (0x253f bytes) 2024-09-16T13:28:57.355 INFO:teuthology.orchestra.run.smithi076.stdout:1949 writesame 0x34f79fa thru 0x3501af7 data_size 0x3(0xa0fe bytes) 2024-09-16T13:28:57.357 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:57.356+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:57.371 INFO:teuthology.orchestra.run.smithi076.stdout:1950 write 0x27941ff thru 0x279fbe0 (0xb9e2 bytes) 2024-09-16T13:28:57.382 INFO:teuthology.orchestra.run.smithi076.stdout:1951 read 0xc1462de thru 0xc15369f (0xd3c2 bytes) 2024-09-16T13:28:57.382 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:57.382 INFO:teuthology.orchestra.run.smithi076.stdout:1956 punch from 0xc0b9d3 to 0xc10ef2, (0x551f bytes) 2024-09-16T13:28:57.382 INFO:teuthology.orchestra.run.smithi076.stdout:1957 writesame 0x1643d9d thru 0x1645193 data_size 0x1(0x13f7 bytes) 2024-09-16T13:28:57.390 INFO:teuthology.orchestra.run.smithi076.stdout:1958 trunc from 0xd174b3c to 0xb095cd1 2024-09-16T13:28:57.418 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:28:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:28:57.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 52.126476050999997, 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.419 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:28:57.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.425125854000001, 2024-09-16T13:28:57.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:28:57.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:28:57.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:28:57.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.425003457999999, 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:28:57.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:28:57.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:28:57.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424891070000001, 2024-09-16T13:28:57.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:28:57.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424828255999998, 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:28:57.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424680825999999, 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:28:57.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.453 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424665642999997, 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:28:57.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:28:57.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:28:57.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424644596, 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.459 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:28:57.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:28:57.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:28:57.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424557116000003, 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424534606999998, 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.470 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:28:57.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.475 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424488279999998, 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:57.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:28:57.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424321523000003, 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:28:57.481 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:28:57.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:28:57.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.424306758, 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.487 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:57.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.492 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.078984441000003, 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.493 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:28:57.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:28:57.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.498 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.078946762999998, 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:28:57.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:28:57.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:57.504 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.078923949999997, 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:28:57.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.078838736999998, 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.510 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:28:57.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.772259812000001, 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.516 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:28:57.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:28:57.520 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.746351577999999, 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:28:57.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.524 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.677586423000001, 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:28:57.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:28:57.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:28:57.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:28:57.529 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:28:57.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:28:57.531 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:57.529+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:57.629 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:28:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:28:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:28:57.629 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:28:57.631 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:57.630+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:57.635 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:57.635 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:57.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:28:57.651 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:57.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:57.666 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:57.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:57.681 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:57.681 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:57.697 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:57.697 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:57.712 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:57.713 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:57.728 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:57.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:28:57.732 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:57.731+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:57.744 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:28:57.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:28:57.751 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:57.751 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:57.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:28:57.845 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:57.843+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:57.900 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:57.900 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:57.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:28:57.950 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:57.948+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:58.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:28:58.035 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:58.034+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:58.049 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.050 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.059 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:28:58.115 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T13:28:58.128 INFO:teuthology.orchestra.run.smithi076.stdout:1959 read 0x81593a3 thru 0x8162c28 (0x9886 bytes) 2024-09-16T13:28:58.128 INFO:teuthology.orchestra.run.smithi076.stdout:1960 punch from 0xa4a0279 to 0xa4aaf01, (0xac88 bytes) 2024-09-16T13:28:58.128 INFO:teuthology.orchestra.run.smithi076.stdout:1962 write 0x66981d8 thru 0x66a69a0 (0xe7c9 bytes) 2024-09-16T13:28:58.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T13:28:58.136 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:28:58.134+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:28:58.138 INFO:teuthology.orchestra.run.smithi076.stdout:1963 read 0x439c52b thru 0x43a317c (0x6c52 bytes) 2024-09-16T13:28:58.138 INFO:teuthology.orchestra.run.smithi076.stdout:1964 writesame 0xed8ccb thru 0xee852f data_size 0x1(0xf865 bytes) 2024-09-16T13:28:58.160 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:58.160 INFO:teuthology.orchestra.run.smithi076.stdout:1967 read 0x19da963 thru 0x19e3992 (0x9030 bytes) 2024-09-16T13:28:58.161 INFO:teuthology.orchestra.run.smithi076.stdout:1968 punch from 0xacfe0e5 to 0xad09094, (0xafaf bytes) 2024-09-16T13:28:58.161 INFO:teuthology.orchestra.run.smithi076.stdout:1969 write 0x5bf6ff3 thru 0x5bfc7b5 (0x57c3 bytes) 2024-09-16T13:28:58.173 INFO:teuthology.orchestra.run.smithi076.stdout:1976 write 0x58495e1 thru 0x584e5cb (0x4feb bytes) 2024-09-16T13:28:58.183 INFO:teuthology.orchestra.run.smithi076.stdout:1977 read 0xad338bc thru 0xad3e346 (0xaa8b bytes) 2024-09-16T13:28:58.183 INFO:teuthology.orchestra.run.smithi076.stdout:1981 write 0x3998a71 thru 0x399ec76 (0x6206 bytes) 2024-09-16T13:28:58.184 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.184 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:28:58.208 INFO:teuthology.orchestra.run.smithi076.stdout:1982 trunc from 0xb095cd1 to 0x3cf0206 2024-09-16T13:28:58.233 INFO:teuthology.orchestra.run.smithi076.stdout:1983 clone 15 order 25 su 524288 sc 11 2024-09-16T13:28:58.236 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:58.235+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:58.324 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.324 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:28:58.338 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:58.336+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:58.438 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:28:58.436+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:28:58.471 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.472 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:28:58.539 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:58.538+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:58.578 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.578 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:28:58.669 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:58.668+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:58.722 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.723 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:28:58.741 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:58.740+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:58.842 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:58.840+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:58.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:28:58.964 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:58.962+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:59.017 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.017 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:28:59.067 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:28:59.065+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:28:59.125 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T13:28:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:28:59.142 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T13:28:59.144 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:59.142+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:59.172 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.172 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:28:59.243 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-09-16T13:28:59.244 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:59.242+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:59.287 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.287 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.296 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:28:59.380 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:59.378+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:59.434 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.434 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:28:59.484 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:59.482+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:59.491 INFO:teuthology.orchestra.run.smithi076.stdout:1984 write 0x6d8c60d thru 0x6d8ee32 (0x2826 bytes) 2024-09-16T13:28:59.498 INFO:teuthology.orchestra.run.smithi076.stdout:1985 write 0x67123b5 thru 0x6712ebf (0xb0b bytes) 2024-09-16T13:28:59.512 INFO:teuthology.orchestra.run.smithi076.stdout:1986 write 0x9992257 thru 0x9997090 (0x4e3a bytes) 2024-09-16T13:28:59.513 INFO:teuthology.orchestra.run.smithi076.stdout:1987 write 0x7965e67 thru 0x796a81f (0x49b9 bytes) 2024-09-16T13:28:59.539 INFO:teuthology.orchestra.run.smithi076.stdout:1988 write 0xdf347ed thru 0xdf3827d (0x3a91 bytes) 2024-09-16T13:28:59.546 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:28:59.544+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:28:59.555 INFO:teuthology.orchestra.run.smithi076.stdout:1989 read 0x693586e thru 0x693c167 (0x68fa bytes) 2024-09-16T13:28:59.555 INFO:teuthology.orchestra.run.smithi076.stdout:1990 writesame 0x783d63d thru 0x783fa18 data_size 0x44(0x23dc bytes) 2024-09-16T13:28:59.561 INFO:teuthology.orchestra.run.smithi076.stdout:1991 write 0x50f53e5 thru 0x510327b (0xde97 bytes) 2024-09-16T13:28:59.577 INFO:teuthology.orchestra.run.smithi076.stdout:1992 read 0xcb083c1 thru 0xcb0ea6c (0x66ac bytes) 2024-09-16T13:28:59.577 INFO:teuthology.orchestra.run.smithi076.stdout:1993 writesame 0x1684686 thru 0x1688bed data_size 0x4(0x4568 bytes) 2024-09-16T13:28:59.592 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.592 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.593 INFO:teuthology.orchestra.run.smithi076.stdout:1994 write 0x6526b92 thru 0x6526f94 (0x403 bytes) 2024-09-16T13:28:59.603 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:28:59.606 INFO:teuthology.orchestra.run.smithi076.stdout:1995 writesame 0x166868c thru 0x1672c82 data_size 0x1(0xa5f7 bytes) 2024-09-16T13:28:59.646 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:28:59.645+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:28:59.681 INFO:teuthology.orchestra.run.smithi076.stdout:1996 write 0x421ed59 thru 0x42298a1 (0xab49 bytes) 2024-09-16T13:28:59.685 INFO:teuthology.orchestra.run.smithi076.stdout:1997 read 0x9862d10 thru 0x986745a (0x474b bytes) 2024-09-16T13:28:59.685 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.685 INFO:teuthology.orchestra.run.smithi076.stdout:1999 read 0xcd7a8da thru 0xcd82bc8 (0x82ef bytes) 2024-09-16T13:28:59.686 INFO:teuthology.orchestra.run.smithi076.stdout:2000 write 0xada100b thru 0xada7ed7 (0x6ecd bytes) 2024-09-16T13:28:59.687 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.693 INFO:teuthology.orchestra.run.smithi076.stdout:2003 trunc from 0xdf3827e to 0x9a4e5f7 2024-09-16T13:28:59.717 INFO:teuthology.orchestra.run.smithi076.stdout:2005 read 0x809582 thru 0x80efe9 (0x5a68 bytes) 2024-09-16T13:28:59.718 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.718 INFO:teuthology.orchestra.run.smithi076.stdout:2007 read 0x50569cb thru 0x5060e7d (0xa4b3 bytes) 2024-09-16T13:28:59.721 INFO:teuthology.orchestra.run.smithi076.stdout:2008 trunc from 0x9a4e5f7 to 0x75504f7 2024-09-16T13:28:59.734 INFO:teuthology.orchestra.run.smithi076.stdout:2009 trunc from 0x75504f7 to 0x96a8bbc 2024-09-16T13:28:59.740 INFO:teuthology.orchestra.run.smithi076.stdout:2014 read 0x7fba9a8 thru 0x7fbd53b (0x2b94 bytes) 2024-09-16T13:28:59.741 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.741 INFO:teuthology.orchestra.run.smithi076.stdout:2017 writesame 0x6b3c472 thru 0x6b3c996 data_size 0x1(0x525 bytes) 2024-09-16T13:28:59.744 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.747 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:28:59.745+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:28:59.752 INFO:teuthology.orchestra.run.smithi076.stdout:2019 write 0xcc0354b thru 0xcc04468 (0xf1e bytes) 2024-09-16T13:28:59.752 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.752 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.753 INFO:teuthology.orchestra.run.smithi076.stdout:2021 write 0x6f2bd5a thru 0x6f38f0d (0xd1b4 bytes) 2024-09-16T13:28:59.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:28:59.759 INFO:teuthology.orchestra.run.smithi076.stdout:2022 trunc from 0xcc04469 to 0xe407192 2024-09-16T13:28:59.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:28:59.764 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.764 INFO:teuthology.orchestra.run.smithi076.stdout:2025 writesame 0xc97e0f1 thru 0xc98757f data_size 0x7(0x948f bytes) 2024-09-16T13:28:59.768 INFO:teuthology.orchestra.run.smithi076.stdout:2031 punch from 0x67b55f3 to 0x67c46e7, (0xf0f4 bytes) 2024-09-16T13:28:59.768 INFO:teuthology.orchestra.run.smithi076.stdout:2032 read 0xb408ee9 thru 0xb40fa28 (0x6b40 bytes) 2024-09-16T13:28:59.768 INFO:teuthology.orchestra.run.smithi076.stdout:2033 punch from 0x89b9b68 to 0x89c3c9a, (0xa132 bytes) 2024-09-16T13:28:59.768 INFO:teuthology.orchestra.run.smithi076.stdout:2034 write 0xc109be7 thru 0xc117212 (0xd62c bytes) 2024-09-16T13:28:59.771 INFO:teuthology.orchestra.run.smithi076.stdout:2035 read 0xadbf2b5 thru 0xadbfa7c (0x7c8 bytes) 2024-09-16T13:28:59.787 INFO:teuthology.orchestra.run.smithi076.stdout:2036 trunc from 0xe407192 to 0x1a34a8e 2024-09-16T13:28:59.818 INFO:teuthology.orchestra.run.smithi076.stdout:2038 write 0xded5790 thru 0xdedf503 (0x9d74 bytes) 2024-09-16T13:28:59.820 INFO:teuthology.orchestra.run.smithi076.stdout:2039 read 0x876d145 thru 0x876e457 (0x1313 bytes) 2024-09-16T13:28:59.820 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.820 INFO:teuthology.orchestra.run.smithi076.stdout:2042 punch from 0xbe96d65 to 0xbe9ba50, (0x4ceb bytes) 2024-09-16T13:28:59.820 INFO:teuthology.orchestra.run.smithi076.stdout:2046 writesame 0x42ee928 thru 0x42f07ef data_size 0x4(0x1ec8 bytes) 2024-09-16T13:28:59.822 INFO:teuthology.orchestra.run.smithi076.stdout:2048 write 0x77b6c2a thru 0x77bd0b7 (0x648e bytes) 2024-09-16T13:28:59.824 INFO:teuthology.orchestra.run.smithi076.stdout:2049 punch from 0x15d48b3 to 0x15d5a44, (0x1191 bytes) 2024-09-16T13:28:59.828 INFO:teuthology.orchestra.run.smithi076.stdout:2051 trunc from 0xdedf504 to 0xa1bacec 2024-09-16T13:28:59.836 INFO:teuthology.orchestra.run.smithi076.stdout:2053 read 0x7bfd01a thru 0x7c05512 (0x84f9 bytes) 2024-09-16T13:28:59.836 INFO:teuthology.orchestra.run.smithi076.stdout:2054 read 0x8536465 thru 0x853822b (0x1dc7 bytes) 2024-09-16T13:28:59.836 INFO:teuthology.orchestra.run.smithi076.stdout:2056 writesame 0x6d62fbc thru 0x6d72f3f data_size 0x5c(0xff84 bytes) 2024-09-16T13:28:59.840 INFO:teuthology.orchestra.run.smithi076.stdout:2057 trunc from 0xa1bacec to 0xbce3c35 2024-09-16T13:28:59.846 INFO:teuthology.orchestra.run.smithi076.stdout:2059 write 0x65770fe thru 0x6578801 (0x1704 bytes) 2024-09-16T13:28:59.848 INFO:teuthology.orchestra.run.smithi076.stdout:2060 read 0x66e605e thru 0x66f4376 (0xe319 bytes) 2024-09-16T13:28:59.848 INFO:teuthology.orchestra.run.smithi076.stdout:2061 writesame 0x68a355d thru 0x68a7709 data_size 0x17(0x41ad bytes) 2024-09-16T13:28:59.848 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:28:59.846+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:28:59.849 INFO:teuthology.orchestra.run.smithi076.stdout:2062 writesame 0x953e4da thru 0x95477d3 data_size 0x6(0x92fa bytes) 2024-09-16T13:28:59.857 INFO:teuthology.orchestra.run.smithi076.stdout:2064 punch from 0x26dfa55 to 0x26e174d, (0x1cf8 bytes) 2024-09-16T13:28:59.857 INFO:teuthology.orchestra.run.smithi076.stdout:2065 read 0x703f69c thru 0x7048bbb (0x9520 bytes) 2024-09-16T13:28:59.857 INFO:teuthology.orchestra.run.smithi076.stdout:2067 read 0x568f033 thru 0x569b914 (0xc8e2 bytes) 2024-09-16T13:28:59.857 INFO:teuthology.orchestra.run.smithi076.stdout:2069 read 0xa4e162a thru 0xa4ee574 (0xcf4b bytes) 2024-09-16T13:28:59.858 INFO:teuthology.orchestra.run.smithi076.stdout:2072 write 0x4ec3d26 thru 0x4ed382a (0xfb05 bytes) 2024-09-16T13:28:59.863 INFO:teuthology.orchestra.run.smithi076.stdout:2073 punch from 0x108168d to 0x1087ee9, (0x685c bytes) 2024-09-16T13:28:59.863 INFO:teuthology.orchestra.run.smithi076.stdout:2075 write 0xce758 thru 0xdc090 (0xd939 bytes) 2024-09-16T13:28:59.867 INFO:teuthology.orchestra.run.smithi076.stdout:2077 write 0x4d68bf3 thru 0x4d6ff34 (0x7342 bytes) 2024-09-16T13:28:59.869 INFO:teuthology.orchestra.run.smithi076.stdout:2078 write 0x99a73de thru 0x99b5455 (0xe078 bytes) 2024-09-16T13:28:59.873 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.873 INFO:teuthology.orchestra.run.smithi076.stdout:2080 read 0x5036066 thru 0x5038afe (0x2a99 bytes) 2024-09-16T13:28:59.873 INFO:teuthology.orchestra.run.smithi076.stdout:2081 writesame 0xa2f3057 thru 0xa2f7006 data_size 0x8(0x3fb0 bytes) 2024-09-16T13:28:59.877 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:28:59.877 INFO:teuthology.orchestra.run.smithi076.stdout:2083 writesame 0x2881182 thru 0x2890cb4 data_size 0x1(0xfb33 bytes) 2024-09-16T13:28:59.884 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:28:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:28:59.906 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:28:59.906 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:28:59.908 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:28:59.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:28:59.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:28:59.924 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:28:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:28:59.939 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:28:59.940 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:28:59.949 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:28:59.947+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:28:59.956 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:28:59.956 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:28:59.972 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:28:59.972 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:28:59.989 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:28:59.989 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:00.007 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:00.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:00.011 INFO:teuthology.orchestra.run.smithi076.stdout:2084 trunc from 0xbce3c35 to 0x309ff8a 2024-09-16T13:29:00.018 INFO:teuthology.orchestra.run.smithi076.stdout:2085 write 0x3eda50 thru 0x3f4f7a (0x752b bytes) 2024-09-16T13:29:00.020 INFO:teuthology.orchestra.run.smithi076.stdout:2086 write 0x2a2233e thru 0x2a24057 (0x1d1a bytes) 2024-09-16T13:29:00.021 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.024 INFO:teuthology.orchestra.run.smithi076.stdout:2090 trunc from 0x309ff8a to 0x5bab085 2024-09-16T13:29:00.030 INFO:teuthology.orchestra.run.smithi076.stdout:2091 read 0xfd4c25 thru 0xfdc52c (0x7908 bytes) 2024-09-16T13:29:00.030 INFO:teuthology.orchestra.run.smithi076.stdout:2093 punch from 0x42e686d to 0x42f2dc7, (0xc55a bytes) 2024-09-16T13:29:00.030 INFO:teuthology.orchestra.run.smithi076.stdout:2095 punch from 0x15dfa71 to 0x15ea3e0, (0xa96f bytes) 2024-09-16T13:29:00.041 INFO:teuthology.orchestra.run.smithi076.stdout:2097 writesame 0xa72f144 thru 0xa739510 data_size 0x1(0xa3cd bytes) 2024-09-16T13:29:00.045 INFO:teuthology.orchestra.run.smithi076.stdout:2099 trunc from 0xa739511 to 0x8e2a208 2024-09-16T13:29:00.050 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:00.048+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:00.051 INFO:teuthology.orchestra.run.smithi076.stdout:2100 read 0x47d2d39 thru 0x47dd770 (0xaa38 bytes) 2024-09-16T13:29:00.052 INFO:teuthology.orchestra.run.smithi076.stdout:2101 write 0x806e5b3 thru 0x806e6fb (0x149 bytes) 2024-09-16T13:29:00.059 INFO:teuthology.orchestra.run.smithi076.stdout:2102 writesame 0xa4d04f1 thru 0xa4d1fcd data_size 0xd(0x1add bytes) 2024-09-16T13:29:00.060 INFO:teuthology.orchestra.run.smithi076.stdout:2103 read 0x479a8e0 thru 0x479ab67 (0x288 bytes) 2024-09-16T13:29:00.060 INFO:teuthology.orchestra.run.smithi076.stdout:2104 writesame 0x75a2a4 thru 0x761c48 data_size 0x13(0x79a5 bytes) 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.42021815, 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:00.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:00.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:00.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:00.068 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.329687745000001, 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:00.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:00.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:00.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:00.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3536 4.3 e38/32 4:c24635eb:::rbd_data.11a976a8123d.0000000000000000:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.320527046999999, 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018037721, 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.078 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739177+0000", 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739179+0000", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.316e-06 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742916+0000", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037375210000000002 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742931+0000", 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5413999999999998e-05 2024-09-16T13:29:00.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742941+0000", 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4290000000000007e-06 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742984+0000", 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3022000000000002e-05 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.742999+0000", 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5279000000000001e-05 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757171+0000", 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014172034999999999 2024-09-16T13:29:00.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757215+0000", 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3705000000000003e-05 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.302558574999999, 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:00.083 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:00.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:00.087 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.886796423, 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.816460141, 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6960 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.677034097, 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018031049, 2024-09-16T13:29:00.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6960 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382670+0000", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382671+0000", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382672+0000", 2024-09-16T13:29:00.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.601e-06 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382678+0000", 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382681+0000", 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400597+0000", 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017916527000000002 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400640+0000", 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2394000000000001e-05 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400701+0000", 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1334000000000006e-05 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6962 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.677019424000001, 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018139466, 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6962 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382685+0000", 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382687+0000", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382694+0000", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7800000000000003e-06 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382697+0000", 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:00.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400722+0000", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018024563 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400763+0000", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1003e-05 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400824+0000", 2024-09-16T13:29:00.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1496999999999998e-05 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6968 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 720896~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.677003491000001, 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018252758000000001, 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.105 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6968 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382701+0000", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382703+0000", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382712+0000", 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9600000000000006e-06 2024-09-16T13:29:00.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382715+0000", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5640000000000001e-06 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400844+0000", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018129125999999999 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400893+0000", 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8869000000000003e-05 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400954+0000", 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.02e-05 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6970 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 983040~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676984168000001, 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018372506, 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.109 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6970 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382720+0000", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382722+0000", 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:29:00.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382728+0000", 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5029999999999997e-06 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382751+0000", 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3697e-05 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.400983+0000", 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018231872999999999 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401028+0000", 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5148999999999998e-05 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401093+0000", 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4222999999999998e-05 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.113 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6984 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1245184~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676850156, 2024-09-16T13:29:00.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018365686999999999, 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6984 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382854+0000", 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382855+0000", 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.06e-07 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382858+0000", 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382868+0000", 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9450000000000005e-06 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382877+0000", 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3829999999999998e-06 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401119+0000", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018241384999999999 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401160+0000", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1393999999999997e-05 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401220+0000", 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9558999999999997e-05 2024-09-16T13:29:00.118 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6986 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1507328~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676821328999999, 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018458407, 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.119 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6986 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382883+0000", 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382886+0000", 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.159e-06 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382892+0000", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3690000000000003e-06 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382896+0000", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9899999999999999e-06 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401242+0000", 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018346475000000001 2024-09-16T13:29:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401282+0000", 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401341+0000", 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8906999999999997e-05 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6990 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2293760~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676804279999999, 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018563608999999998, 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.124 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6990 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382900+0000", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382902+0000", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382909+0000", 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382913+0000", 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.552e-06 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401365+0000", 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018452196000000001 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401405+0000", 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0525999999999998e-05 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401464+0000", 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8226000000000001e-05 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.128 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:6991 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:7 [sparse-read 2555904~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676787429000001, 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018669403000000001, 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6991 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.129 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382917+0000", 2024-09-16T13:29:00.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382919+0000", 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.671e-06 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382926+0000", 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7460000000000003e-06 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382929+0000", 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401486+0000", 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018556492000000001 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401525+0000", 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9338999999999997e-05 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401586+0000", 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1124000000000006e-05 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7002 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.67677115, 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018778276, 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.133 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7002 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382933+0000", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.58e-07 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382942+0000", 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4149999999999999e-06 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382949+0000", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2840000000000001e-06 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382953+0000", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401609+0000", 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018656005999999999 2024-09-16T13:29:00.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401650+0000", 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1077e-05 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401711+0000", 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1245000000000001e-05 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7003 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.676747183, 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018881326, 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7003 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382957+0000", 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:29:00.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382959+0000", 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.61e-06 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382977+0000", 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7997999999999999e-05 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.382985+0000", 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7929999999999998e-06 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401733+0000", 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018748602 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401775+0000", 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1751000000000001e-05 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401838+0000", 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3188000000000006e-05 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7028 4.15 4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.675807492000001, 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018078818999999999, 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.142 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7028 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383897+0000", 2024-09-16T13:29:00.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383898+0000", 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0249999999999999e-06 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383902+0000", 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.357e-06 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383912+0000", 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0356999999999999e-05 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383918+0000", 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3290000000000001e-06 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401861+0000", 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017943489999999999 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401916+0000", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4722999999999998e-05 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.401976+0000", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537999999999998e-05 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:00.146 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.373338394999999, 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:00.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:00.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:00.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:00.151 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.363249128, 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:00.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:00.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2164604939999997, 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.156 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:00.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:00.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.162 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.165 INFO:teuthology.orchestra.run.smithi076.stdout:2106 read 0x9e2d0b0 thru 0x9e3528e (0x81df bytes) 2024-09-16T13:29:00.165 INFO:teuthology.orchestra.run.smithi076.stdout:2107 write 0xf56315 thru 0xf65016 (0xed02 bytes) 2024-09-16T13:29:00.165 INFO:teuthology.orchestra.run.smithi076.stdout:2108 flatten 2024-09-16T13:29:00.165 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.165 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.166 INFO:teuthology.orchestra.run.smithi076.stdout:2111 read 0x1597096 thru 0x15a4585 (0xd4f0 bytes) 2024-09-16T13:29:00.166 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.166 INFO:teuthology.orchestra.run.smithi076.stdout:2113 read 0x3305d9d thru 0x33086dc (0x2940 bytes) 2024-09-16T13:29:00.166 INFO:teuthology.orchestra.run.smithi076.stdout:2114 writesame 0x7f731ec thru 0x7f7ae70 data_size 0x1(0x7c85 bytes) 2024-09-16T13:29:00.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:00.167 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:00.161+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:00.167 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T13:29:00.168 INFO:teuthology.orchestra.run.smithi076.stdout:2115 writesame 0xe2f14e2 thru 0xe2f8f83 data_size 0xb(0x7aa2 bytes) 2024-09-16T13:29:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:2116 read 0xd8d50e thru 0xd8f7e1 (0x22d4 bytes) 2024-09-16T13:29:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:2117 writesame 0x92070f7 thru 0x920dd12 data_size 0x44(0x6c1c bytes) 2024-09-16T13:29:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:2119 write 0x141c42c thru 0x1426e9f (0xaa74 bytes) 2024-09-16T13:29:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:2121 read 0x4e48d6f thru 0x4e54c27 (0xbeb9 bytes) 2024-09-16T13:29:00.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T13:29:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:2122 trunc from 0xe2f8f84 to 0x140af7e 2024-09-16T13:29:00.171 INFO:teuthology.orchestra.run.smithi076.stdout:2123 read 0x9fbb24 thru 0x9fefaa (0x3487 bytes) 2024-09-16T13:29:00.174 INFO:teuthology.orchestra.run.smithi076.stdout:2125 write 0x3b73506 thru 0x3b73f2c (0xa27 bytes) 2024-09-16T13:29:00.176 INFO:teuthology.orchestra.run.smithi076.stdout:2128 punch from 0x1125a73 to 0x112f291, (0x981e bytes) 2024-09-16T13:29:00.176 INFO:teuthology.orchestra.run.smithi076.stdout:2129 write 0x1bd7021 thru 0x1be0f63 (0x9f43 bytes) 2024-09-16T13:29:00.178 INFO:teuthology.orchestra.run.smithi076.stdout:2130 writesame 0x25677db thru 0x2569752 data_size 0x8(0x1f78 bytes) 2024-09-16T13:29:00.179 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.179 INFO:teuthology.orchestra.run.smithi076.stdout:2133 punch from 0x3517131 to 0x3517907, (0x7d6 bytes) 2024-09-16T13:29:00.196 INFO:teuthology.orchestra.run.smithi076.stdout:2134 writesame 0xc928c8d thru 0xc92e5af data_size 0x1(0x5923 bytes) 2024-09-16T13:29:00.197 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.209 INFO:teuthology.orchestra.run.smithi076.stdout:2138 trunc from 0xc92e5b0 to 0x1fb168b 2024-09-16T13:29:00.219 INFO:teuthology.orchestra.run.smithi076.stdout:2140 punch from 0x15d9525 to 0x15e62da, (0xcdb5 bytes) 2024-09-16T13:29:00.240 INFO:teuthology.orchestra.run.smithi076.stdout:2141 writesame 0xd501afe thru 0xd503104 data_size 0x1(0x1607 bytes) 2024-09-16T13:29:00.241 INFO:teuthology.orchestra.run.smithi076.stdout:2142 read 0xb6c5186 thru 0xb6c9605 (0x4480 bytes) 2024-09-16T13:29:00.241 INFO:teuthology.orchestra.run.smithi076.stdout:2143 writesame 0xa357256 thru 0xa362977 data_size 0xb(0xb722 bytes) 2024-09-16T13:29:00.247 INFO:teuthology.orchestra.run.smithi076.stdout:2145 read 0x82c9608 thru 0x82d80fa (0xeaf3 bytes) 2024-09-16T13:29:00.247 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.247 INFO:teuthology.orchestra.run.smithi076.stdout:2148 punch from 0xa0ffb7a to 0xa1032c5, (0x374b bytes) 2024-09-16T13:29:00.247 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.248 INFO:teuthology.orchestra.run.smithi076.stdout:2151 trunc from 0xd503105 to 0xc41055b 2024-09-16T13:29:00.259 INFO:teuthology.orchestra.run.smithi076.stdout:2152 write 0x5c099b8 thru 0x5c17b5c (0xe1a5 bytes) 2024-09-16T13:29:00.263 INFO:teuthology.orchestra.run.smithi076.stdout:2153 punch from 0x1da0f37 to 0x1da7366, (0x642f bytes) 2024-09-16T13:29:00.263 INFO:teuthology.orchestra.run.smithi076.stdout:2154 read 0xa0856b9 thru 0xa092254 (0xcb9c bytes) 2024-09-16T13:29:00.264 INFO:teuthology.orchestra.run.smithi076.stdout:2156 read 0x9e8bda6 thru 0x9e90010 (0x426b bytes) 2024-09-16T13:29:00.264 INFO:teuthology.orchestra.run.smithi076.stdout:2157 write 0x5185b24 thru 0x5186ad1 (0xfae bytes) 2024-09-16T13:29:00.264 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:00.262+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:00.265 INFO:teuthology.orchestra.run.smithi076.stdout:2158 write 0xae54402 thru 0xae5d914 (0x9513 bytes) 2024-09-16T13:29:00.267 INFO:teuthology.orchestra.run.smithi076.stdout:2159 write 0xa398336 thru 0xa3a39d8 (0xb6a3 bytes) 2024-09-16T13:29:00.271 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.272 INFO:teuthology.orchestra.run.smithi076.stdout:2161 punch from 0x322567f to 0x322fa3f, (0xa3c0 bytes) 2024-09-16T13:29:00.272 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:00.272 INFO:teuthology.orchestra.run.smithi076.stdout:2163 writesame 0x28aa88e thru 0x28b52f1 data_size 0x14(0xaa64 bytes) 2024-09-16T13:29:00.274 INFO:teuthology.orchestra.run.smithi076.stdout:2164 punch from 0x657a63b to 0x657c506, (0x1ecb bytes) 2024-09-16T13:29:00.274 INFO:teuthology.orchestra.run.smithi076.stdout:2165 clone 16 order 24 su 131072 sc 5 2024-09-16T13:29:00.309 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.309 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:00.310 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:00.310 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.318 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:00.365 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:00.363+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:00.417 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.417 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:00.417 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:00.417 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:00.417 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:00.467 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:00.465+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:00.565 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:00.566 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.875636786000001, 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.567 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:00.568 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:00.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.571 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:00.573 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:00.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:00.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:00.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:00.578 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:00.578 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.578 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.578 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.866065308, 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.579 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7494 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 10944512~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.583 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.018676844000002, 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036610563999999998, 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7494 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:29:00.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543979+0000", 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.585 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543980+0000", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.85e-07 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543983+0000", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543985+0000", 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.586 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580303+0000", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036318691 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580359+0000", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5072999999999999e-05 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580590+0000", 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023103699999999999 2024-09-16T13:29:00.587 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.018668896000001, 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.588 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:00.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:00.592 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.018656625999999, 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.596 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.018645781, 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:29:00.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.600 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.018636366999999, 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:00.601 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:00.602 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.603 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:00.604 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8154 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272404906999999, 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036367568000000003, 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.605 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8154 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290251+0000", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290252+0000", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4799999999999997e-07 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290255+0000", 2024-09-16T13:29:00.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9239999999999999e-06 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290257+0000", 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7859999999999999e-06 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326341+0000", 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036084789999999999 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.608 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326384+0000", 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509999999999998e-05 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326619+0000", 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023450899999999999 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.609 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272397234, 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.610 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.611 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:29:00.613 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272388888999998, 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:29:00.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:29:00.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272378195000002, 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.618 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:29:00.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:29:00.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:00.622 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272369207000001, 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:00.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272361466, 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:00.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272352906999998, 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.631 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.635 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272344029999999, 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:00.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:00.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272330165, 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:00.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.644 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272321697999999, 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:00.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.272312894999999, 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:00.649 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:00.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6488428370000001, 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:00.654 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:00.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.6356141560000008, 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:00.660 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:00.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:00.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.664 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.665 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:00.666 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:00.663+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:00.763 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.763 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:00.763 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:00.763 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.766 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:00.764+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:00.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:00.866 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:00.865+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:00.869 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.869 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:00.869 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:00.869 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:00.869 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:00.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:00.967 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:00.965+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:00.978 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.686731780999999, 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.985381584000002, 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.985259188000001, 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.985146800000003, 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:29:00.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:00.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.985083985999999, 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:01.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:01.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984936556000001, 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:29:01.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:29:01.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984921372999999, 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:01.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:01.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:01.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:01.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:01.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984900326000002, 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.019 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:01.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:01.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:01.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:01.024 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984812845999997, 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.025 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.026 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:01.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:01.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:01.033 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984790337, 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:01.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:01.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.98474401, 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.039 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:01.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:01.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.045 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984577252999998, 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:01.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:01.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.984562488000002, 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:01.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:01.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:01.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:01.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:01.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:01.060 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.639240170999997, 2024-09-16T13:29:01.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:01.061 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.061 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.061 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.061 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:01.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:01.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.639202492999999, 2024-09-16T13:29:01.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:01.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:01.074 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.639179679999998, 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:01.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:01.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:01.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.080 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.639094467, 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.081 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:01.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.332515541999999, 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:01.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.090 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.306607308, 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:01.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:01.095 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.237842152999999, 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:01.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:01.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:01.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:01.100 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:01.101 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:01.100+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:01.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:01.137 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T13:29:01.143 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:29:01.153 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T13:29:01.220 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:01.220 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:01.220 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:01.220 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:01.220 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:01.218+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:01.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:01.317 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:01.315+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:01.337 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-09-16T13:29:01.403 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:01.401+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:01.421 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:01.421 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:01.421 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:01.421 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:01.421 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:01.423 INFO:teuthology.orchestra.run.smithi076.stdout:2167 writesame 0x177f234 thru 0x178bdec data_size 0x1(0xcbb9 bytes) 2024-09-16T13:29:01.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:01.504 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:01.502+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:01.532 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:01.532 INFO:teuthology.orchestra.run.smithi076.stdout:2170 punch from 0x8a40a7c to 0x8a4dd5e, (0xd2e2 bytes) 2024-09-16T13:29:01.532 INFO:teuthology.orchestra.run.smithi076.stdout:2171 writesame 0x8d35b34 thru 0x8d4064f data_size 0x5e(0xab1c bytes) 2024-09-16T13:29:01.545 INFO:teuthology.orchestra.run.smithi076.stdout:2173 punch from 0x466022e to 0x46654b4, (0x5286 bytes) 2024-09-16T13:29:01.545 INFO:teuthology.orchestra.run.smithi076.stdout:2174 read 0x723fabe thru 0x7248ae6 (0x9029 bytes) 2024-09-16T13:29:01.551 INFO:teuthology.orchestra.run.smithi076.stdout:2175 write 0xc9e7206 thru 0xc9ed7d0 (0x65cb bytes) 2024-09-16T13:29:01.557 INFO:teuthology.orchestra.run.smithi076.stdout:2176 write 0xaa5ebc0 thru 0xaa61ab7 (0x2ef8 bytes) 2024-09-16T13:29:01.568 INFO:teuthology.orchestra.run.smithi076.stdout:2177 write 0x4eb56e7 thru 0x4ec0d95 (0xb6af bytes) 2024-09-16T13:29:01.589 INFO:teuthology.orchestra.run.smithi076.stdout:2178 write 0x9179456 thru 0x9180c0e (0x77b9 bytes) 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:01.593 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.423046021000001, 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:01.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:01.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:01.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:01.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.417570456, 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.599 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:01.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.657243649000002, 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.605 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:01.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:01.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:01.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.640705524000001, 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8145 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298124954999999, 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041847961000000003, 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.615 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8145 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290587+0000", 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.03e-07 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290589+0000", 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.086e-06 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290596+0000", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1509999999999998e-06 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290600+0000", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.692e-06 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332033+0000", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041433316999999997 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332120+0000", 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6847000000000004e-05 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332434+0000", 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031446499999999999 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8146 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:01.619 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298107994999999, 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042182996, 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8146 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290604+0000", 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290605+0000", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290612+0000", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.686e-06 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290616+0000", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332462+0000", 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041846176999999998 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332534+0000", 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2398999999999996e-05 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332786+0000", 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025215300000000002 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8147 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298091923000001, 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042474660999999997, 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8147 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290620+0000", 2024-09-16T13:29:01.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290621+0000", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290627+0000", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9020000000000004e-06 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290632+0000", 2024-09-16T13:29:01.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6609999999999999e-06 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332812+0000", 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042180278000000002 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332854+0000", 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1857999999999998e-05 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333094+0000", 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000239905 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298075642000001, 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.628 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:01.632 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298058663999999, 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:01.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:29:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.636 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298042074000001, 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.637 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:01.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298019729, 2024-09-16T13:29:01.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:01.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.298001749000001, 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.646 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:01.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:01.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.297985945000001, 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.650 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:01.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.297970383999999, 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:01.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.297952670000001, 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.659 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:01.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.297926087, 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:01.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:01.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.456372173999998, 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.258932603000002, 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:01.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:01.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:01.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:01.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:29:01.679 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.248433855999998, 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:01.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:29:01.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.684055997, 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:01.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:01.688 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:01.688 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:01.689 INFO:teuthology.orchestra.run.smithi076.stdout:2179 punch from 0x39dc5ca to 0x39e213a, (0x5b70 bytes) 2024-09-16T13:29:01.689 INFO:teuthology.orchestra.run.smithi076.stdout:2182 write 0xc5fc3cf thru 0xc60944f (0xd081 bytes) 2024-09-16T13:29:01.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2183 write 0x372b63f thru 0x372d4af (0x1e71 bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2184 write 0x27f5fa5 thru 0x2802203 (0xc25f bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2190 write 0x9d31841 thru 0x9d407f4 (0xefb4 bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2192 read 0xb554a00 thru 0xb55742d (0x2a2e bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2194 read 0x9b1a1dd thru 0x9b2734c (0xd170 bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2195 write 0x54962c5 thru 0x5498756 (0x2492 bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2196 write 0xd8d1819 thru 0xd8d7a22 (0x620a bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2197 read 0x2ad26f0 thru 0x2adfbfa (0xd50b bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2198 punch from 0x645727c to 0x645a0e8, (0x2e6c bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2199 punch from 0x57305d4 to 0x5738f12, (0x893e bytes) 2024-09-16T13:29:01.692 INFO:teuthology.orchestra.run.smithi076.stdout:2200 punch from 0x69be985 to 0x69c91c9, (0xa844 bytes) 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:2201 read 0x278a8ea thru 0x278e698 (0x3daf bytes) 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:2203 read 0x715c77 thru 0x722015 (0xc39f bytes) 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:2205 writesame 0xcc1e8df thru 0xcc1f5ba data_size 0x2(0xcdc bytes) 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:2207 writesame 0x6e4e59c thru 0x6e5af8f data_size 0x37(0xc9f4 bytes) 2024-09-16T13:29:01.693 INFO:teuthology.orchestra.run.smithi076.stdout:2211 trunc from 0xd8d7a23 to 0x6e1ee68 2024-09-16T13:29:01.693 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:01.687+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:01.694 INFO:teuthology.orchestra.run.smithi076.stdout:2212 write 0xc3510b1 thru 0xc3604c2 (0xf412 bytes) 2024-09-16T13:29:01.697 INFO:teuthology.orchestra.run.smithi076.stdout:2213 write 0xb84f345 thru 0xb854820 (0x54dc bytes) 2024-09-16T13:29:01.701 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:01.701 INFO:teuthology.orchestra.run.smithi076.stdout:2216 clone 17 order 20 su 8192 sc 15 2024-09-16T13:29:01.790 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:01.788+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:01.790 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:01.791 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:01.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:01.890 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:01.889+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:01.937 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:01.937 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:01.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:02.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:02.030 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:02.028+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:02.089 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.643521213, 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:29:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:29:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.471122037000001, 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.236050163000002, 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:29:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:29:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.383350041, 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.362286133000001, 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.058213094999999, 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:29:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:29:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.953619273000001, 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:29:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:29:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:29:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21707 4.14 e46/32 4:28551aaf:::rbd_data.11a98415fd69.0000000000000000:head v 46'74, mlcod=45'72)", 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.948876040000002, 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015037457000000001, 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.136989+0000", 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5300000000000003e-07 2024-09-16T13:29:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137531+0000", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054107700000000005 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137543+0000", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2156e-05 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137548+0000", 2024-09-16T13:29:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.463e-06 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137896+0000", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034767700000000001 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.137917+0000", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1523999999999999e-05 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.151679+0000", 2024-09-16T13:29:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013761997 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.152026+0000", 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034681000000000002 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.947075632000001, 2024-09-16T13:29:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23819 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.270174570999998, 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018047839999999999, 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23819 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815691+0000", 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.054e-06 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815696+0000", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1749999999999996e-06 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815708+0000", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1446e-05 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815715+0000", 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T13:29:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824348+0000", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086330599999999997 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824416+0000", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8213999999999994e-05 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824679+0000", 2024-09-16T13:29:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026342200000000003 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826830+0000", 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002150749 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833601+0000", 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067705100000000004 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833698+0000", 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7188000000000005e-05 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833738+0000", 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0173000000000003e-05 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23820 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.270143916999999, 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018255289000000001, 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23820 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:29:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815725+0000", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7840000000000001e-06 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815732+0000", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9040000000000003e-06 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815737+0000", 2024-09-16T13:29:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5549999999999996e-06 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825011+0000", 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092747530000000002 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825098+0000", 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6287999999999998e-05 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825475+0000", 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037692200000000002 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826843+0000", 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001368403 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833748+0000", 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069050989999999996 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833884+0000", 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013585 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833976+0000", 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2219999999999995e-05 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23821 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.270122524000001, 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018354471000000001, 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23821 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815743+0000", 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.46e-07 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815747+0000", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889999999999999e-06 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815756+0000", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-06 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815760+0000", 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825875+0000", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010114911000000001 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825919+0000", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3572000000000002e-05 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826099+0000", 2024-09-16T13:29:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018034099999999999 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827140+0000", 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010402899999999999 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833988+0000", 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068483349999999997 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834065+0000", 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7305999999999995e-05 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834097+0000", 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1492000000000003e-05 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25655 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.484185216, 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014517317, 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25655 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601680+0000", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601681+0000", 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6169999999999999e-06 2024-09-16T13:29:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601688+0000", 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.294e-06 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601703+0000", 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4124999999999999e-05 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601709+0000", 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1090000000000003e-06 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601751+0000", 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2293000000000003e-05 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.601810+0000", 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8786000000000003e-05 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.602757+0000", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094687500000000002 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616128+0000", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013370885000000001 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616135+0000", 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9300000000000003e-06 2024-09-16T13:29:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616171+0000", 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5196e-05 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.616197+0000", 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.455315873, 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.446439294999999, 2024-09-16T13:29:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.286445085, 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:29:02.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.286412056, 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.176 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:29:02.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:02.183 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.275048161999999, 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:29:02.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:28298 4.7 4:e7bc7339:::rbd_data.11a9320547a2.0000000000000003:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.785808940999999, 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033840752000000002, 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 28298 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.188 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300057+0000", 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300062+0000", 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2820000000000004e-06 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300075+0000", 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2975e-05 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300081+0000", 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9830000000000001e-06 2024-09-16T13:29:02.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300108+0000", 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7155999999999999e-05 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300156+0000", 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7311000000000002e-05 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:02.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.302362+0000", 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002206253 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333731+0000", 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031369085999999997 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333794+0000", 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2712999999999994e-05 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333852+0000", 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8013999999999997e-05 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333897+0000", 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4728999999999998e-05 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:28298 4.7 e51/32 ondisk, result = 0)", 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.777400817, 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025439566, 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308465+0000", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308472+0000", 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0480000000000001e-06 2024-09-16T13:29:02.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308494+0000", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1712000000000002e-05 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308504+0000", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4509999999999998e-06 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333763+0000", 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025259193999999999 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333785+0000", 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2497000000000002e-05 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333904+0000", 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118263 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:02.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:02.200 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:02.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:02.201 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:29:02.202 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T13:29:02.202 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:02.202 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:02.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T13:29:02.204 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:02.204 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:02.205 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:02.205 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:02.205 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:02.199+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:02.205 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone16 from 0xc3604c3 (overlap 0x6e1ee68) to 0x5818b86 2024-09-16T13:29:02.206 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:02.206 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:02.221 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:02.238 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:02.238 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:02.255 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:02.256 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:02.286 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:02.286 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:02.301 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:02.300+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:02.305 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-09-16T13:29:02.329 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:02.329 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:02.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:02.426 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:02.424+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:02.484 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:02.485 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:02.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:02.503 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:02.501+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:02.535 INFO:teuthology.orchestra.run.smithi076.stdout:2217 trunc from 0xc3604c3 to 0x4960f2b 2024-09-16T13:29:02.604 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:02.602+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:02.633 INFO:teuthology.orchestra.run.smithi076.stdout:2218 trunc from 0x4960f2b to 0xb43687a 2024-09-16T13:29:02.639 INFO:teuthology.orchestra.run.smithi076.stdout:2219 write 0x2837ddd thru 0x283d9bf (0x5be3 bytes) 2024-09-16T13:29:02.670 INFO:teuthology.orchestra.run.smithi076.stdout:2220 read 0x4de3ba5 thru 0x4deceb0 (0x930c bytes) 2024-09-16T13:29:02.670 INFO:teuthology.orchestra.run.smithi076.stdout:2221 write 0x3ba30f thru 0x3c1fd0 (0x7cc2 bytes) 2024-09-16T13:29:02.679 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:02.679 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:02.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:02.736 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:02.734+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:02.789 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:02.867 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:02.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:02.869 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:02.866+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:02.900 INFO:teuthology.orchestra.run.smithi076.stdout:2222 read 0x2f065cf thru 0x2f06b94 (0x5c6 bytes) 2024-09-16T13:29:02.901 INFO:teuthology.orchestra.run.smithi076.stdout:2224 write 0x18c4eb thru 0x197c3e (0xb754 bytes) 2024-09-16T13:29:02.949 INFO:teuthology.orchestra.run.smithi076.stdout:2225 punch from 0xd9d279 to 0xd9def6, (0xc7d bytes) 2024-09-16T13:29:02.949 INFO:teuthology.orchestra.run.smithi076.stdout:2229 write 0x430a111 thru 0x430ec17 (0x4b07 bytes) 2024-09-16T13:29:02.960 INFO:teuthology.orchestra.run.smithi076.stdout:2230 trunc from 0xb43687a to 0x2b98dc3 2024-09-16T13:29:02.968 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:02.967+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:03.036 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:03.041 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.041 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.042 INFO:teuthology.orchestra.run.smithi076.stdout:2232 write 0x3750b10 thru 0x3757aa7 (0x6f98 bytes) 2024-09-16T13:29:03.048 INFO:teuthology.orchestra.run.smithi076.stdout:2233 punch from 0x33d34e8 to 0x33e0316, (0xce2e bytes) 2024-09-16T13:29:03.048 INFO:teuthology.orchestra.run.smithi076.stdout:2234 punch from 0x1df2196 to 0x1df6d42, (0x4bac bytes) 2024-09-16T13:29:03.048 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:03.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:03.053 INFO:teuthology.orchestra.run.smithi076.stdout:2236 trunc from 0x3757aa8 to 0x799f899 2024-09-16T13:29:03.064 INFO:teuthology.orchestra.run.smithi076.stdout:2238 write 0x91c80df thru 0x91d652c (0xe44e bytes) 2024-09-16T13:29:03.070 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:03.068+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:03.084 INFO:teuthology.orchestra.run.smithi076.stdout:2239 trunc from 0x91d652d to 0x37bf92b 2024-09-16T13:29:03.109 INFO:teuthology.orchestra.run.smithi076.stdout:2240 trunc from 0x37bf92b to 0xa86e778 2024-09-16T13:29:03.113 INFO:teuthology.orchestra.run.smithi076.stdout:2241 punch from 0x20be051 to 0x20beb09, (0xab8 bytes) 2024-09-16T13:29:03.115 INFO:teuthology.orchestra.run.smithi076.stdout:2242 trunc from 0xa86e778 to 0x8d72c6c 2024-09-16T13:29:03.121 INFO:teuthology.orchestra.run.smithi076.stdout:2243 punch from 0x38e8eb3 to 0x38f7401, (0xe54e bytes) 2024-09-16T13:29:03.121 INFO:teuthology.orchestra.run.smithi076.stdout:2244 punch from 0x22bb6f1 to 0x22c3b37, (0x8446 bytes) 2024-09-16T13:29:03.121 INFO:teuthology.orchestra.run.smithi076.stdout:2245 writesame 0x60a1491 thru 0x60ae8f7 data_size 0x7d(0xd467 bytes) 2024-09-16T13:29:03.133 INFO:teuthology.orchestra.run.smithi076.stdout:2246 writesame 0x469c130 thru 0x46a885b data_size 0x1c(0xc72c bytes) 2024-09-16T13:29:03.149 INFO:teuthology.orchestra.run.smithi076.stdout:2247 punch from 0x40c1ba7 to 0x40cedc5, (0xd21e bytes) 2024-09-16T13:29:03.149 INFO:teuthology.orchestra.run.smithi076.stdout:2248 read 0x853467d thru 0x8538cff (0x4683 bytes) 2024-09-16T13:29:03.150 INFO:teuthology.orchestra.run.smithi076.stdout:2250 punch from 0x36af6b1 to 0x36b1294, (0x1be3 bytes) 2024-09-16T13:29:03.150 INFO:teuthology.orchestra.run.smithi076.stdout:2251 write 0x3d2de42 thru 0x3d3d55b (0xf71a bytes) 2024-09-16T13:29:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T13:29:03.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T13:29:03.170 INFO:teuthology.orchestra.run.smithi076.stdout:2252 read 0x7ff1453 thru 0x7ff9e96 (0x8a44 bytes) 2024-09-16T13:29:03.170 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:03.168+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:03.174 INFO:teuthology.orchestra.run.smithi076.stdout:2253 trunc from 0x8d72c6c to 0x5341123 2024-09-16T13:29:03.175 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.175 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:03.189 INFO:teuthology.orchestra.run.smithi076.stdout:2254 trunc from 0x5341123 to 0x7c82ae5 2024-09-16T13:29:03.194 INFO:teuthology.orchestra.run.smithi076.stdout:2255 flatten 2024-09-16T13:29:03.273 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:03.271+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:03.334 INFO:teuthology.orchestra.run.smithi076.stdout:2257 write 0x33b669c thru 0x33be222 (0x7b87 bytes) 2024-09-16T13:29:03.335 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.335 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.343 INFO:teuthology.orchestra.run.smithi076.stdout:2258 write 0x489f0bb thru 0x48ac262 (0xd1a8 bytes) 2024-09-16T13:29:03.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:03.348 INFO:teuthology.orchestra.run.smithi076.stdout:2259 trunc from 0x7c82ae5 to 0x89f947a 2024-09-16T13:29:03.362 INFO:teuthology.orchestra.run.smithi076.stdout:2262 write 0xcaeb2d6 thru 0xcaf008c (0x4db7 bytes) 2024-09-16T13:29:03.373 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:03.372+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:03.376 INFO:teuthology.orchestra.run.smithi076.stdout:2263 write 0xd5b3ff6 thru 0xd5bf14c (0xb157 bytes) 2024-09-16T13:29:03.392 INFO:teuthology.orchestra.run.smithi076.stdout:2266 trunc from 0xd5bf14d to 0x925edb5 2024-09-16T13:29:03.403 INFO:teuthology.orchestra.run.smithi076.stdout:2268 writesame 0x7840d07 thru 0x78507d5 data_size 0xd(0xfacf bytes) 2024-09-16T13:29:03.416 INFO:teuthology.orchestra.run.smithi076.stdout:2269 write 0x2b3defe thru 0x2b4ccb9 (0xedbc bytes) 2024-09-16T13:29:03.419 INFO:teuthology.orchestra.run.smithi076.stdout:2270 punch from 0x40663bc to 0x40668e2, (0x526 bytes) 2024-09-16T13:29:03.422 INFO:teuthology.orchestra.run.smithi076.stdout:2271 trunc from 0x925edb5 to 0xb928952 2024-09-16T13:29:03.433 INFO:teuthology.orchestra.run.smithi076.stdout:2272 writesame 0xc8e1d4a thru 0xc8e52a9 data_size 0x3d(0x3560 bytes) 2024-09-16T13:29:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:2273 writesame 0xce801b6 thru 0xce8fe4b data_size 0x27(0xfc96 bytes) 2024-09-16T13:29:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:2275 read 0x3bcc980 thru 0x3bd4aea (0x816b bytes) 2024-09-16T13:29:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:2276 write 0x6d3c8a0 thru 0x6d47405 (0xab66 bytes) 2024-09-16T13:29:03.449 INFO:teuthology.orchestra.run.smithi076.stdout:2277 write 0x3157337 thru 0x31589c0 (0x168a bytes) 2024-09-16T13:29:03.450 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.450 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.451 INFO:teuthology.orchestra.run.smithi076.stdout:2280 write 0x7d2f7f thru 0x7ddb33 (0xabb5 bytes) 2024-09-16T13:29:03.453 INFO:teuthology.orchestra.run.smithi076.stdout:2282 punch from 0x4bfe354 to 0x4c06b51, (0x87fd bytes) 2024-09-16T13:29:03.453 INFO:teuthology.orchestra.run.smithi076.stdout:2283 clone 18 order 20 su 16384 sc 5 2024-09-16T13:29:03.460 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:03.474 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:03.473+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:03.557 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.558 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:03.575 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:03.573+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:03.664 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.664 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:03.676 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:03.674+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:03.770 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.771 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.777 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:03.775+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:03.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:03.876 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.876 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.877 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:03.875+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:03.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:03.978 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:03.976+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:03.983 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:03.983 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:03.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:04.078 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:04.077+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:04.130 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:04.130 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:04.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:04.161 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T13:29:04.177 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T13:29:04.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:29:04.179 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:04.177+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:04.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:29:04.277 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:04.277 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:04.279 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:04.278+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:04.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:04.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:04.369 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-09-16T13:29:04.380 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:04.379+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:04.386 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:04.386 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:04.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:04.436 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:04.453 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:04.453 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:04.469 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:04.470 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:04.481 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:04.479+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:04.485 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:04.485 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:04.493 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:04.493 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:04.499 INFO:teuthology.orchestra.run.smithi076.stdout:2284 writesame 0xe927f4 thru 0xe9a1bb data_size 0x24(0x79c8 bytes) 2024-09-16T13:29:04.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:04.504 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:04.505 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:04.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:04.536 INFO:teuthology.orchestra.run.smithi076.stdout:2285 read 0xa2e60e thru 0xa2e836 (0x229 bytes) 2024-09-16T13:29:04.536 INFO:teuthology.orchestra.run.smithi076.stdout:2286 read 0x163d675 thru 0x164878f (0xb11b bytes) 2024-09-16T13:29:04.537 INFO:teuthology.orchestra.run.smithi076.stdout:2288 read 0x427f0aa thru 0x4286bdf (0x7b36 bytes) 2024-09-16T13:29:04.538 INFO:teuthology.orchestra.run.smithi076.stdout:2289 read 0x6e148b6 thru 0x6e18252 (0x399d bytes) 2024-09-16T13:29:04.538 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:04.538 INFO:teuthology.orchestra.run.smithi076.stdout:2292 writesame 0x41c6e50 thru 0x41cb1fd data_size 0x1(0x43ae bytes) 2024-09-16T13:29:04.555 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:04.555 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:04.570 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:04.570 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:29:04.573 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:04.573 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:04.578 INFO:teuthology.orchestra.run.smithi076.stdout:2293 read 0x580e595 thru 0x5817778 (0x91e4 bytes) 2024-09-16T13:29:04.581 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:04.580+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:04.584 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-16T13:29:04.156525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769863186,"num_objects":801,"num_object_clones":197,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":262,"num_read":55814,"num_read_kb":289225,"num_write":6641,"num_write_kb":170679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":22962992,"num_keys_recovered":36,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5067,"ondisk_log_size":5067,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":2159464,"kb_used_data":36368,"kb_used_omap":11,"kb_used_meta":2122996,"kb_avail":658443416,"statfs":{"total":676457349120,"available":674246057984,"internally_reserved":0,"allocated":37240832,"data_stored":337390287,"data_compressed":7369600,"data_compressed_allocated":25456640,"data_compressed_original":330896738,"omap_allocated":11928,"internal_metadata":2173948264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":105052780,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":4914,"num_read_kb":34053,"num_write":328,"num_write_kb":12819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032187"},"pg_stats":[{"pgid":"4.1f","version":"62'137","reported_seq":1028,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.127695+0000","last_change":"2024-09-16T13:28:28.989963+0000","last_active":"2024-09-16T13:29:03.127695+0000","last_peered":"2024-09-16T13:29:03.127695+0000","last_clean":"2024-09-16T13:29:03.127695+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:29:03.127695+0000","last_undegraded":"2024-09-16T13:29:03.127695+0000","last_fullsized":"2024-09-16T13:29:03.127695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089281099999999997,"stat_sum":{"num_bytes":48729220,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":1865,"num_read_kb":7483,"num_write":146,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"62'228","reported_seq":9364,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.132925+0000","last_change":"2024-09-16T13:28:55.915873+0000","last_active":"2024-09-16T13:29:03.132925+0000","last_peered":"2024-09-16T13:29:03.132925+0000","last_clean":"2024-09-16T13:29:03.132925+0000","last_became_active":"2024-09-16T13:28:55.915295+0000","last_became_peered":"2024-09-16T13:28:55.915295+0000","last_unstale":"2024-09-16T13:29:03.132925+0000","last_undegraded":"2024-09-16T13:29:03.132925+0000","last_fullsized":"2024-09-16T13:29:03.132925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:06.241381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041021900000000002,"stat_sum":{"num_bytes":52564245,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":6,"num_read":10305,"num_read_kb":17442,"num_write":362,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1345764,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"58'126","reported_seq":399,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225933+0000","last_change":"2024-09-16T13:28:28.984841+0000","last_active":"2024-09-16T13:28:58.225933+0000","last_peered":"2024-09-16T13:28:58.225933+0000","last_clean":"2024-09-16T13:28:58.225933+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:28:58.225933+0000","last_undegraded":"2024-09-16T13:28:58.225933+0000","last_fullsized":"2024-09-16T13:28:58.225933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071631099999999997,"stat_sum":{"num_bytes":14043935,"num_objects":20,"num_object_clones":6,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":9,"num_read":609,"num_read_kb":2663,"num_write":117,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"62'90","reported_seq":595,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.285646+0000","last_change":"2024-09-16T13:28:28.981596+0000","last_active":"2024-09-16T13:29:02.285646+0000","last_peered":"2024-09-16T13:29:02.285646+0000","last_clean":"2024-09-16T13:29:02.285646+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:29:02.285646+0000","last_undegraded":"2024-09-16T13:29:02.285646+0000","last_fullsized":"2024-09-16T13:29:02.285646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030921799999999999,"stat_sum":{"num_bytes":6850058,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":582,"num_read_kb":7925,"num_write":100,"num_write_kb":3153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"58'65","reported_seq":192,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225727+0000","last_change":"2024-09-16T13:28:28.979438+0000","last_active":"2024-09-16T13:28:58.225727+0000","last_peered":"2024-09-16T13:28:58.225727+0000","last_clean":"2024-09-16T13:28:58.225727+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:28:58.225727+0000","last_undegraded":"2024-09-16T13:28:58.225727+0000","last_fullsized":"2024-09-16T13:28:58.225727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056234500000000001,"stat_sum":{"num_bytes":8243780,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":7,"num_read":127,"num_read_kb":2535,"num_write":63,"num_write_kb":1844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"58'112","reported_seq":1198,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.152218+0000","last_change":"2024-09-16T13:28:28.979457+0000","last_active":"2024-09-16T13:28:59.152218+0000","last_peered":"2024-09-16T13:28:59.152218+0000","last_clean":"2024-09-16T13:28:59.152218+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:28:59.152218+0000","last_undegraded":"2024-09-16T13:28:59.152218+0000","last_fullsized":"2024-09-16T13:28:59.152218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058420599999999998,"stat_sum":{"num_bytes":16408257,"num_objects":24,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":4,"num_read":1185,"num_read_kb":5560,"num_write":126,"num_write_kb":1054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"59'84","reported_seq":737,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.153743+0000","last_change":"2024-09-16T13:28:28.983849+0000","last_active":"2024-09-16T13:28:59.153743+0000","last_peered":"2024-09-16T13:28:59.153743+0000","last_clean":"2024-09-16T13:28:59.153743+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:28:59.153743+0000","last_undegraded":"2024-09-16T13:28:59.153743+0000","last_fullsized":"2024-09-16T13:28:59.153743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041077799999999999,"stat_sum":{"num_bytes":27253196,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":12,"num_read":657,"num_read_kb":1417,"num_write":76,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"58'103","reported_seq":1197,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221911+0000","last_change":"2024-09-16T13:28:28.979247+0000","last_active":"2024-09-16T13:28:58.221911+0000","last_peered":"2024-09-16T13:28:58.221911+0000","last_clean":"2024-09-16T13:28:58.221911+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:28:58.221911+0000","last_undegraded":"2024-09-16T13:28:58.221911+0000","last_fullsized":"2024-09-16T13:28:58.221911+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039287600000000001,"stat_sum":{"num_bytes":24780927,"num_objects":29,"num_object_clones":10,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":7,"num_read":1663,"num_read_kb":9891,"num_write":104,"num_write_kb":6267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"57'68","reported_seq":626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.160193+0000","last_change":"2024-09-16T13:28:53.162614+0000","last_active":"2024-09-16T13:28:58.160193+0000","last_peered":"2024-09-16T13:28:58.160193+0000","last_clean":"2024-09-16T13:28:58.160193+0000","last_became_active":"2024-09-16T13:28:53.086651+0000","last_became_peered":"2024-09-16T13:28:53.086651+0000","last_unstale":"2024-09-16T13:28:58.160193+0000","last_undegraded":"2024-09-16T13:28:58.160193+0000","last_fullsized":"2024-09-16T13:28:58.160193+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:50.595475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080754000000000004,"stat_sum":{"num_bytes":10070264,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":8,"num_read":802,"num_read_kb":12939,"num_write":75,"num_write_kb":7602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":7625158,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"62'129","reported_seq":493,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.147068+0000","last_change":"2024-09-16T13:28:30.062793+0000","last_active":"2024-09-16T13:29:03.147068+0000","last_peered":"2024-09-16T13:29:03.147068+0000","last_clean":"2024-09-16T13:29:03.147068+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:29:03.147068+0000","last_undegraded":"2024-09-16T13:29:03.147068+0000","last_fullsized":"2024-09-16T13:29:03.147068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039817100000000001,"stat_sum":{"num_bytes":7804752,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":823,"num_read_kb":1783,"num_write":137,"num_write_kb":1885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"59'290","reported_seq":1415,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.131601+0000","last_change":"2024-09-16T13:28:55.918381+0000","last_active":"2024-09-16T13:28:59.131601+0000","last_peered":"2024-09-16T13:28:59.131601+0000","last_clean":"2024-09-16T13:28:59.131601+0000","last_became_active":"2024-09-16T13:28:55.917981+0000","last_became_peered":"2024-09-16T13:28:55.917981+0000","last_unstale":"2024-09-16T13:28:59.131601+0000","last_undegraded":"2024-09-16T13:28:59.131601+0000","last_fullsized":"2024-09-16T13:28:59.131601+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:03.006738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054,"stat_sum":{"num_bytes":62610518,"num_objects":34,"num_object_clones":8,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":13,"num_read":2246,"num_read_kb":16605,"num_write":497,"num_write_kb":8894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8196910,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"58'193","reported_seq":1431,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.218466+0000","last_change":"2024-09-16T13:28:28.984368+0000","last_active":"2024-09-16T13:28:58.218466+0000","last_peered":"2024-09-16T13:28:58.218466+0000","last_clean":"2024-09-16T13:28:58.218466+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:28:58.218466+0000","last_undegraded":"2024-09-16T13:28:58.218466+0000","last_fullsized":"2024-09-16T13:28:58.218466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000271229,"stat_sum":{"num_bytes":5405397,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":13,"num_read":2422,"num_read_kb":4871,"num_write":235,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"57'83","reported_seq":544,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.112460+0000","last_change":"2024-09-16T13:28:28.979874+0000","last_active":"2024-09-16T13:28:58.112460+0000","last_peered":"2024-09-16T13:28:58.112460+0000","last_clean":"2024-09-16T13:28:58.112460+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:28:58.112460+0000","last_undegraded":"2024-09-16T13:28:58.112460+0000","last_fullsized":"2024-09-16T13:28:58.112460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000816496,"stat_sum":{"num_bytes":34385391,"num_objects":26,"num_object_clones":9,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":10,"num_read":464,"num_read_kb":9900,"num_write":78,"num_write_kb":2149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"62'375","reported_seq":897,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.012944+0000","last_change":"2024-09-16T13:28:28.979687+0000","last_active":"2024-09-16T13:29:03.012944+0000","last_peered":"2024-09-16T13:29:03.012944+0000","last_clean":"2024-09-16T13:29:03.012944+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:29:03.012944+0000","last_undegraded":"2024-09-16T13:29:03.012944+0000","last_fullsized":"2024-09-16T13:29:03.012944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035537999999999998,"stat_sum":{"num_bytes":4587631,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":5,"num_read":2546,"num_read_kb":4636,"num_write":585,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"58'174","reported_seq":2131,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225394+0000","last_change":"2024-09-16T13:28:55.918299+0000","last_active":"2024-09-16T13:28:58.225394+0000","last_peered":"2024-09-16T13:28:58.225394+0000","last_clean":"2024-09-16T13:28:58.225394+0000","last_became_active":"2024-09-16T13:28:55.917756+0000","last_became_peered":"2024-09-16T13:28:55.917756+0000","last_unstale":"2024-09-16T13:28:58.225394+0000","last_undegraded":"2024-09-16T13:28:58.225394+0000","last_fullsized":"2024-09-16T13:28:58.225394+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:17:58.519127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035811000000000003,"stat_sum":{"num_bytes":7410330,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":10,"num_read":2217,"num_read_kb":3736,"num_write":214,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":535741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"58'151","reported_seq":486,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.223903+0000","last_change":"2024-09-16T13:28:28.983418+0000","last_active":"2024-09-16T13:28:58.223903+0000","last_peered":"2024-09-16T13:28:58.223903+0000","last_clean":"2024-09-16T13:28:58.223903+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:28:58.223903+0000","last_undegraded":"2024-09-16T13:28:58.223903+0000","last_fullsized":"2024-09-16T13:28:58.223903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044577700000000001,"stat_sum":{"num_bytes":2371619,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":5,"num_read":1270,"num_read_kb":3214,"num_write":218,"num_write_kb":2261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.148006+0000","last_change":"2024-09-16T13:28:53.082764+0000","last_active":"2024-09-16T13:29:02.148006+0000","last_peered":"2024-09-16T13:29:02.148006+0000","last_clean":"2024-09-16T13:29:02.148006+0000","last_became_active":"2024-09-16T13:28:53.082485+0000","last_became_peered":"2024-09-16T13:28:53.082485+0000","last_unstale":"2024-09-16T13:29:02.148006+0000","last_undegraded":"2024-09-16T13:29:02.148006+0000","last_fullsized":"2024-09-16T13:29:02.148006+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:48:45.781441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"58'181","reported_seq":503,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.222397+0000","last_change":"2024-09-16T13:28:28.979658+0000","last_active":"2024-09-16T13:28:58.222397+0000","last_peered":"2024-09-16T13:28:58.222397+0000","last_clean":"2024-09-16T13:28:58.222397+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:28:58.222397+0000","last_undegraded":"2024-09-16T13:28:58.222397+0000","last_fullsized":"2024-09-16T13:28:58.222397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066082900000000004,"stat_sum":{"num_bytes":12755214,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":8,"num_read":808,"num_read_kb":11943,"num_write":253,"num_write_kb":7508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":55,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.153088+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:02.153088+0000","last_peered":"2024-09-16T13:29:02.153088+0000","last_clean":"2024-09-16T13:29:02.153088+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:29:02.153088+0000","last_undegraded":"2024-09-16T13:29:02.153088+0000","last_fullsized":"2024-09-16T13:29:02.153088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"58'143","reported_seq":906,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221855+0000","last_change":"2024-09-16T13:28:55.915810+0000","last_active":"2024-09-16T13:28:58.221855+0000","last_peered":"2024-09-16T13:28:58.221855+0000","last_clean":"2024-09-16T13:28:58.221855+0000","last_became_active":"2024-09-16T13:28:55.915373+0000","last_became_peered":"2024-09-16T13:28:55.915373+0000","last_unstale":"2024-09-16T13:28:58.221855+0000","last_undegraded":"2024-09-16T13:28:58.221855+0000","last_fullsized":"2024-09-16T13:28:58.221855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:47.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043631399999999998,"stat_sum":{"num_bytes":10481989,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1057,"num_read_kb":2678,"num_write":205,"num_write_kb":1397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1074310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.147957+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:29:02.147957+0000","last_peered":"2024-09-16T13:29:02.147957+0000","last_clean":"2024-09-16T13:29:02.147957+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:29:02.147957+0000","last_undegraded":"2024-09-16T13:29:02.147957+0000","last_fullsized":"2024-09-16T13:29:02.147957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"62'303","reported_seq":2104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.132909+0000","last_change":"2024-09-16T13:28:30.062555+0000","last_active":"2024-09-16T13:29:03.132909+0000","last_peered":"2024-09-16T13:29:03.132909+0000","last_clean":"2024-09-16T13:29:03.132909+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:29:03.132909+0000","last_undegraded":"2024-09-16T13:29:03.132909+0000","last_fullsized":"2024-09-16T13:29:03.132909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000230199,"stat_sum":{"num_bytes":47472852,"num_objects":39,"num_object_clones":10,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":11,"num_read":2749,"num_read_kb":41917,"num_write":453,"num_write_kb":27201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.153172+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:02.153172+0000","last_peered":"2024-09-16T13:29:02.153172+0000","last_clean":"2024-09-16T13:29:02.153172+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:29:02.153172+0000","last_undegraded":"2024-09-16T13:29:02.153172+0000","last_fullsized":"2024-09-16T13:29:02.153172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"62'60","reported_seq":228,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.123959+0000","last_change":"2024-09-16T13:28:28.979887+0000","last_active":"2024-09-16T13:29:03.123959+0000","last_peered":"2024-09-16T13:29:03.123959+0000","last_clean":"2024-09-16T13:29:03.123959+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:29:03.123959+0000","last_undegraded":"2024-09-16T13:29:03.123959+0000","last_fullsized":"2024-09-16T13:29:03.123959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053138000000000003,"stat_sum":{"num_bytes":4694136,"num_objects":22,"num_object_clones":3,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":5,"num_read":155,"num_read_kb":2216,"num_write":70,"num_write_kb":2917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.112518+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:58.112518+0000","last_peered":"2024-09-16T13:28:58.112518+0000","last_clean":"2024-09-16T13:28:58.112518+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:58.112518+0000","last_undegraded":"2024-09-16T13:28:58.112518+0000","last_fullsized":"2024-09-16T13:28:58.112518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":53,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.123775+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:58.123775+0000","last_peered":"2024-09-16T13:28:58.123775+0000","last_clean":"2024-09-16T13:28:58.123775+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:58.123775+0000","last_undegraded":"2024-09-16T13:28:58.123775+0000","last_fullsized":"2024-09-16T13:28:58.123775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"62'101","reported_seq":1764,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.143472+0000","last_change":"2024-09-16T13:28:28.979855+0000","last_active":"2024-09-16T13:29:03.143472+0000","last_peered":"2024-09-16T13:29:03.143472+0000","last_clean":"2024-09-16T13:29:03.143472+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:29:03.143472+0000","last_undegraded":"2024-09-16T13:29:03.143472+0000","last_fullsized":"2024-09-16T13:29:03.143472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039579799999999999,"stat_sum":{"num_bytes":23358961,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1714,"num_read_kb":10101,"num_write":118,"num_write_kb":2153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.152859+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:29:02.152859+0000","last_peered":"2024-09-16T13:29:02.152859+0000","last_clean":"2024-09-16T13:29:02.152859+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:29:02.152859+0000","last_undegraded":"2024-09-16T13:29:02.152859+0000","last_fullsized":"2024-09-16T13:29:02.152859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T13:29:01.144131+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:29:01.144131+0000","last_peered":"2024-09-16T13:29:01.144131+0000","last_clean":"2024-09-16T13:29:01.144131+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:29:01.144131+0000","last_undegraded":"2024-09-16T13:29:01.144131+0000","last_fullsized":"2024-09-16T13:29:01.144131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"62'134","reported_seq":492,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.007684+0000","last_change":"2024-09-16T13:28:28.980209+0000","last_active":"2024-09-16T13:29:03.007684+0000","last_peered":"2024-09-16T13:29:03.007684+0000","last_clean":"2024-09-16T13:29:03.007684+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:29:03.007684+0000","last_undegraded":"2024-09-16T13:29:03.007684+0000","last_fullsized":"2024-09-16T13:29:03.007684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069467499999999996,"stat_sum":{"num_bytes":48269494,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":10,"num_read":373,"num_read_kb":3970,"num_write":162,"num_write_kb":3421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"53'1","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.177088+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:29:02.177088+0000","last_peered":"2024-09-16T13:29:02.177088+0000","last_clean":"2024-09-16T13:29:02.177088+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:29:02.177088+0000","last_undegraded":"2024-09-16T13:29:02.177088+0000","last_fullsized":"2024-09-16T13:29:02.177088+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.152935+0000","last_change":"2024-09-16T13:28:53.086425+0000","last_active":"2024-09-16T13:29:02.152935+0000","last_peered":"2024-09-16T13:29:02.152935+0000","last_clean":"2024-09-16T13:29:02.152935+0000","last_became_active":"2024-09-16T13:28:53.086128+0000","last_became_peered":"2024-09-16T13:28:53.086128+0000","last_unstale":"2024-09-16T13:29:02.152935+0000","last_undegraded":"2024-09-16T13:29:02.152935+0000","last_fullsized":"2024-09-16T13:29:02.152935+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:41:23.848293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"62'127","reported_seq":977,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.092762+0000","last_change":"2024-09-16T13:28:30.062973+0000","last_active":"2024-09-16T13:29:03.092762+0000","last_peered":"2024-09-16T13:29:03.092762+0000","last_clean":"2024-09-16T13:29:03.092762+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:29:03.092762+0000","last_undegraded":"2024-09-16T13:29:03.092762+0000","last_fullsized":"2024-09-16T13:29:03.092762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049121500000000003,"stat_sum":{"num_bytes":34098905,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":1736,"num_read_kb":11762,"num_write":148,"num_write_kb":13518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"53'1","reported_seq":43,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.117425+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:58.117425+0000","last_peered":"2024-09-16T13:28:58.117425+0000","last_clean":"2024-09-16T13:28:58.117425+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:58.117425+0000","last_undegraded":"2024-09-16T13:28:58.117425+0000","last_fullsized":"2024-09-16T13:28:58.117425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:20:30.803710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":52,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.123767+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:58.123767+0000","last_peered":"2024-09-16T13:28:58.123767+0000","last_clean":"2024-09-16T13:28:58.123767+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:58.123767+0000","last_undegraded":"2024-09-16T13:28:58.123767+0000","last_fullsized":"2024-09-16T13:28:58.123767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.177111+0000","last_change":"2024-09-16T13:28:53.093054+0000","last_active":"2024-09-16T13:29:02.177111+0000","last_peered":"2024-09-16T13:29:02.177111+0000","last_clean":"2024-09-16T13:29:02.177111+0000","last_became_active":"2024-09-16T13:28:53.083561+0000","last_became_peered":"2024-09-16T13:28:53.083561+0000","last_unstale":"2024-09-16T13:29:02.177111+0000","last_undegraded":"2024-09-16T13:29:02.177111+0000","last_fullsized":"2024-09-16T13:29:02.177111+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:47:58.719653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"62'157","reported_seq":1134,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.091406+0000","last_change":"2024-09-16T13:28:28.989279+0000","last_active":"2024-09-16T13:29:03.091406+0000","last_peered":"2024-09-16T13:29:03.091406+0000","last_clean":"2024-09-16T13:29:03.091406+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:29:03.091406+0000","last_undegraded":"2024-09-16T13:29:03.091406+0000","last_fullsized":"2024-09-16T13:29:03.091406+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024121900000000001,"stat_sum":{"num_bytes":23522467,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":6,"num_read":1478,"num_read_kb":10265,"num_write":161,"num_write_kb":4481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"58'201","reported_seq":1704,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221673+0000","last_change":"2024-09-16T13:28:55.916522+0000","last_active":"2024-09-16T13:28:58.221673+0000","last_peered":"2024-09-16T13:28:58.221673+0000","last_clean":"2024-09-16T13:28:58.221673+0000","last_became_active":"2024-09-16T13:28:55.915965+0000","last_became_peered":"2024-09-16T13:28:55.915965+0000","last_unstale":"2024-09-16T13:28:58.221673+0000","last_undegraded":"2024-09-16T13:28:58.221673+0000","last_fullsized":"2024-09-16T13:28:58.221673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:07.234321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024448900000000002,"stat_sum":{"num_bytes":7229453,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":7,"num_read":2559,"num_read_kb":4183,"num_write":198,"num_write_kb":2882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1503462,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"58'104","reported_seq":1683,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.220187+0000","last_change":"2024-09-16T13:28:28.979223+0000","last_active":"2024-09-16T13:28:58.220187+0000","last_peered":"2024-09-16T13:28:58.220187+0000","last_clean":"2024-09-16T13:28:58.220187+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:28:58.220187+0000","last_undegraded":"2024-09-16T13:28:58.220187+0000","last_fullsized":"2024-09-16T13:28:58.220187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039311499999999998,"stat_sum":{"num_bytes":46214333,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":9,"num_read":1685,"num_read_kb":12195,"num_write":142,"num_write_kb":4831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"62'147","reported_seq":1687,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.020699+0000","last_change":"2024-09-16T13:28:28.990059+0000","last_active":"2024-09-16T13:29:03.020699+0000","last_peered":"2024-09-16T13:29:03.020699+0000","last_clean":"2024-09-16T13:29:03.020699+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:29:03.020699+0000","last_undegraded":"2024-09-16T13:29:03.020699+0000","last_fullsized":"2024-09-16T13:29:03.020699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086978299999999995,"stat_sum":{"num_bytes":29129227,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1933,"num_read_kb":27564,"num_write":194,"num_write_kb":21936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"62'89","reported_seq":2647,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.087506+0000","last_change":"2024-09-16T13:28:30.062577+0000","last_active":"2024-09-16T13:29:03.087506+0000","last_peered":"2024-09-16T13:29:03.087506+0000","last_clean":"2024-09-16T13:29:03.087506+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:29:03.087506+0000","last_undegraded":"2024-09-16T13:29:03.087506+0000","last_fullsized":"2024-09-16T13:29:03.087506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029103499999999998,"stat_sum":{"num_bytes":53936305,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":8,"num_read":2568,"num_read_kb":18429,"num_write":144,"num_write_kb":15805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"59'200","reported_seq":1914,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.152584+0000","last_change":"2024-09-16T13:28:28.979711+0000","last_active":"2024-09-16T13:28:59.152584+0000","last_peered":"2024-09-16T13:28:59.152584+0000","last_clean":"2024-09-16T13:28:59.152584+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:28:59.152584+0000","last_undegraded":"2024-09-16T13:28:59.152584+0000","last_fullsized":"2024-09-16T13:28:59.152584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071623100000000001,"stat_sum":{"num_bytes":31342720,"num_objects":23,"num_object_clones":3,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3183,"num_read_kb":3374,"num_write":268,"num_write_kb":627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"58'102","reported_seq":908,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.215706+0000","last_change":"2024-09-16T13:28:28.979910+0000","last_active":"2024-09-16T13:28:58.215706+0000","last_peered":"2024-09-16T13:28:58.215706+0000","last_clean":"2024-09-16T13:28:58.215706+0000","last_became_active":"2024-09-16T13:28:26.987044+0000","last_became_peered":"2024-09-16T13:28:26.987044+0000","last_unstale":"2024-09-16T13:28:58.215706+0000","last_undegraded":"2024-09-16T13:28:58.215706+0000","last_fullsized":"2024-09-16T13:28:58.215706+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:11.809480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084175000000000005,"stat_sum":{"num_bytes":25073636,"num_objects":31,"num_object_clones":9,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":12,"num_read":808,"num_read_kb":4578,"num_write":105,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"58'90","reported_seq":289,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.219884+0000","last_change":"2024-09-16T13:28:28.980116+0000","last_active":"2024-09-16T13:28:58.219884+0000","last_peered":"2024-09-16T13:28:58.219884+0000","last_clean":"2024-09-16T13:28:58.219884+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:28:58.219884+0000","last_undegraded":"2024-09-16T13:28:58.219884+0000","last_fullsized":"2024-09-16T13:28:58.219884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010142790000000001,"stat_sum":{"num_bytes":13207063,"num_objects":20,"num_object_clones":6,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":11,"num_read":188,"num_read_kb":5675,"num_write":83,"num_write_kb":3773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'443","reported_seq":1748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.076389+0000","last_change":"2024-09-16T13:28:58.119992+0000","last_active":"2024-09-16T13:29:03.076389+0000","last_peered":"2024-09-16T13:29:03.076389+0000","last_clean":"2024-09-16T13:29:03.076389+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:29:03.076389+0000","last_undegraded":"2024-09-16T13:29:03.076389+0000","last_fullsized":"2024-09-16T13:29:03.076389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074187000000000005,"stat_sum":{"num_bytes":24966524,"num_objects":34,"num_object_clones":11,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":14,"num_read":2971,"num_read_kb":5720,"num_write":701,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":769272799,"num_objects":795,"num_object_clones":197,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":260,"num_read":55748,"num_read_kb":289170,"num_write":6538,"num_write_kb":169049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":22372624,"num_keys_recovered":36,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34095104,"data_stored":348647513,"data_compressed":7760012,"data_compressed_allocated":26775552,"data_compressed_original":344188258,"omap_allocated":0,"internal_metadata":0},"log_size":4990,"ondisk_log_size":4990,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":17628,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":0,"seq":64424509455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306472,"kb_used_data":4956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94065368,"statfs":{"total":96636764160,"available":96322936832,"internally_reserved":0,"allocated":5074944,"data_stored":54199431,"data_compressed":904982,"data_compressed_allocated":3571712,"data_compressed_original":53354303,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305868,"kb_used_data":4348,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94065972,"statfs":{"total":96636764160,"available":96323555328,"internally_reserved":0,"allocated":4452352,"data_stored":29397079,"data_compressed":1206553,"data_compressed_allocated":2756608,"data_compressed_original":28443506,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310648,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061192,"statfs":{"total":96636764160,"available":96318660608,"internally_reserved":0,"allocated":5095424,"data_stored":32708194,"data_compressed":1155483,"data_compressed_allocated":2981888,"data_compressed_original":31551488,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":6,"up_from":14,"seq":60129542162,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":5092,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":5214208,"data_stored":49988340,"data_compressed":1048267,"data_compressed_allocated":3739648,"data_compressed_original":49242994,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303744,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068096,"statfs":{"total":96636764160,"available":96325730304,"internally_reserved":0,"allocated":2277376,"data_stored":10930704,"data_compressed":446790,"data_compressed_allocated":1052672,"data_compressed_original":10317824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.3390000000000004,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":9.4480000000000004,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314388,"kb_used_data":8716,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94057452,"statfs":{"total":96636764160,"available":96314830848,"internally_reserved":0,"allocated":8925184,"data_stored":89037783,"data_compressed":1665396,"data_compressed_allocated":6668288,"data_compressed_original":87748608,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307572,"kb_used_data":6056,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94064268,"statfs":{"total":96636764160,"available":96321810432,"internally_reserved":0,"allocated":6201344,"data_stored":71128756,"data_compressed":942129,"data_compressed_allocated":4685824,"data_compressed_original":70238015,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5435392,"data_stored":70223462,"data_compressed":937009,"data_compressed_allocated":4661248,"data_compressed_original":69689151,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8220672,"data_stored":88719646,"data_compressed":1665480,"data_compressed_allocated":6672384,"data_compressed_original":87756800,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1662976,"data_stored":11205379,"data_compressed":448652,"data_compressed_allocated":1150976,"data_compressed_original":10924032,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4337664,"data_stored":32332766,"data_compressed":1155483,"data_compressed_allocated":2981888,"data_compressed_original":31551488,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3788800,"data_stored":29091552,"data_compressed":1206553,"data_compressed_allocated":2756608,"data_compressed_original":28443506,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":53893885,"data_compressed":904982,"data_compressed_allocated":3571712,"data_compressed_original":53354303,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":49987047,"data_compressed":1055108,"data_compressed_allocated":3756032,"data_compressed_original":49546098,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":13193776,"data_compressed":386745,"data_compressed_allocated":1224704,"data_compressed_original":12922880,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:29:04.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:29:04.587 INFO:teuthology.orchestra.run.smithi076.stdout:2294 writesame 0xd1464d0 thru 0xd148382 data_size 0x1(0x1eb3 bytes) 2024-09-16T13:29:04.590 INFO:teuthology.orchestra.run.smithi076.stdout:2297 read 0x136c743 thru 0x1371f0c (0x57ca bytes) 2024-09-16T13:29:04.592 INFO:teuthology.orchestra.run.smithi076.stdout:2299 read 0xa1fb854 thru 0xa202871 (0x701e bytes) 2024-09-16T13:29:04.592 INFO:teuthology.orchestra.run.smithi076.stdout:2300 punch from 0x3a45776 to 0x3a4a84f, (0x50d9 bytes) 2024-09-16T13:29:04.606 INFO:teuthology.orchestra.run.smithi076.stdout:2301 trunc from 0xd148383 to 0xebfffc 2024-09-16T13:29:04.620 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.328026586, 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626676388999996, 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:29:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:29:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626553993000002, 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626441604999997, 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:29:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:29:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626378791, 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626231361000002, 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076613690000000003, 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990996+0000", 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990997+0000", 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991000+0000", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8890000000000002e-06 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991004+0000", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4050000000000002e-06 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991008+0000", 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.602e-06 2024-09-16T13:29:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994457+0000", 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034495860000000001 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994541+0000", 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3463999999999999e-05 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994675+0000", 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013451199999999999 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997307+0000", 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026320580000000001 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998611+0000", 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001304174 2024-09-16T13:29:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998637+0000", 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.529e-05 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998657+0000", 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0533000000000001e-05 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626216178, 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626195131000003, 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626107650999998, 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626085142000001, 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.626038815000001, 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.625872057999999, 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.625857293000003, 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.280534975999998, 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.280497298, 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.280474484999999, 2024-09-16T13:29:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:04.721 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.280389272000001, 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:04.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:04.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 29.973810347000001, 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:04.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:04.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:04.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 29.947902113000001, 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 29.879136958, 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:04.736 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:04.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:04.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:04.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:04.740 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:04.742 INFO:teuthology.orchestra.run.smithi076.stdout:2305 read 0x7538c2 thru 0x75b9ea (0x8129 bytes) 2024-09-16T13:29:04.742 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:04.742 INFO:teuthology.orchestra.run.smithi076.stdout:2307 clone 19 order 18 su 32768 sc 8 2024-09-16T13:29:04.742 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:04.740+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:04.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:04.842 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:04.842 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:04.842 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:04.842 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:04.842 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:04.841+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:04.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:04.864 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:04.865 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:29:04.875 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-16T13:29:04.156525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769863186,"num_objects":801,"num_object_clones":197,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":262,"num_read":55814,"num_read_kb":289225,"num_write":6641,"num_write_kb":170679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":22962992,"num_keys_recovered":36,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5067,"ondisk_log_size":5067,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":2159464,"kb_used_data":36368,"kb_used_omap":11,"kb_used_meta":2122996,"kb_avail":658443416,"statfs":{"total":676457349120,"available":674246057984,"internally_reserved":0,"allocated":37240832,"data_stored":337390287,"data_compressed":7369600,"data_compressed_allocated":25456640,"data_compressed_original":330896738,"omap_allocated":11928,"internal_metadata":2173948264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":105052780,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":4914,"num_read_kb":34053,"num_write":328,"num_write_kb":12819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032187"},"pg_stats":[{"pgid":"4.1f","version":"62'137","reported_seq":1028,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.127695+0000","last_change":"2024-09-16T13:28:28.989963+0000","last_active":"2024-09-16T13:29:03.127695+0000","last_peered":"2024-09-16T13:29:03.127695+0000","last_clean":"2024-09-16T13:29:03.127695+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:29:03.127695+0000","last_undegraded":"2024-09-16T13:29:03.127695+0000","last_fullsized":"2024-09-16T13:29:03.127695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089281099999999997,"stat_sum":{"num_bytes":48729220,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":1865,"num_read_kb":7483,"num_write":146,"num_write_kb":1253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"62'228","reported_seq":9364,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.132925+0000","last_change":"2024-09-16T13:28:55.915873+0000","last_active":"2024-09-16T13:29:03.132925+0000","last_peered":"2024-09-16T13:29:03.132925+0000","last_clean":"2024-09-16T13:29:03.132925+0000","last_became_active":"2024-09-16T13:28:55.915295+0000","last_became_peered":"2024-09-16T13:28:55.915295+0000","last_unstale":"2024-09-16T13:29:03.132925+0000","last_undegraded":"2024-09-16T13:29:03.132925+0000","last_fullsized":"2024-09-16T13:29:03.132925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:06.241381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041021900000000002,"stat_sum":{"num_bytes":52564245,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":6,"num_read":10305,"num_read_kb":17442,"num_write":362,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1345764,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"58'126","reported_seq":399,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225933+0000","last_change":"2024-09-16T13:28:28.984841+0000","last_active":"2024-09-16T13:28:58.225933+0000","last_peered":"2024-09-16T13:28:58.225933+0000","last_clean":"2024-09-16T13:28:58.225933+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:28:58.225933+0000","last_undegraded":"2024-09-16T13:28:58.225933+0000","last_fullsized":"2024-09-16T13:28:58.225933+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071631099999999997,"stat_sum":{"num_bytes":14043935,"num_objects":20,"num_object_clones":6,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":9,"num_read":609,"num_read_kb":2663,"num_write":117,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"62'90","reported_seq":595,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.285646+0000","last_change":"2024-09-16T13:28:28.981596+0000","last_active":"2024-09-16T13:29:02.285646+0000","last_peered":"2024-09-16T13:29:02.285646+0000","last_clean":"2024-09-16T13:29:02.285646+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:29:02.285646+0000","last_undegraded":"2024-09-16T13:29:02.285646+0000","last_fullsized":"2024-09-16T13:29:02.285646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030921799999999999,"stat_sum":{"num_bytes":6850058,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":582,"num_read_kb":7925,"num_write":100,"num_write_kb":3153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"58'65","reported_seq":192,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225727+0000","last_change":"2024-09-16T13:28:28.979438+0000","last_active":"2024-09-16T13:28:58.225727+0000","last_peered":"2024-09-16T13:28:58.225727+0000","last_clean":"2024-09-16T13:28:58.225727+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:28:58.225727+0000","last_undegraded":"2024-09-16T13:28:58.225727+0000","last_fullsized":"2024-09-16T13:28:58.225727+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056234500000000001,"stat_sum":{"num_bytes":8243780,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":7,"num_read":127,"num_read_kb":2535,"num_write":63,"num_write_kb":1844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"58'112","reported_seq":1198,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.152218+0000","last_change":"2024-09-16T13:28:28.979457+0000","last_active":"2024-09-16T13:28:59.152218+0000","last_peered":"2024-09-16T13:28:59.152218+0000","last_clean":"2024-09-16T13:28:59.152218+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:28:59.152218+0000","last_undegraded":"2024-09-16T13:28:59.152218+0000","last_fullsized":"2024-09-16T13:28:59.152218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058420599999999998,"stat_sum":{"num_bytes":16408257,"num_objects":24,"num_object_clones":3,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":4,"num_read":1185,"num_read_kb":5560,"num_write":126,"num_write_kb":1054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"59'84","reported_seq":737,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.153743+0000","last_change":"2024-09-16T13:28:28.983849+0000","last_active":"2024-09-16T13:28:59.153743+0000","last_peered":"2024-09-16T13:28:59.153743+0000","last_clean":"2024-09-16T13:28:59.153743+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:28:59.153743+0000","last_undegraded":"2024-09-16T13:28:59.153743+0000","last_fullsized":"2024-09-16T13:28:59.153743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041077799999999999,"stat_sum":{"num_bytes":27253196,"num_objects":32,"num_object_clones":7,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":12,"num_read":657,"num_read_kb":1417,"num_write":76,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"58'103","reported_seq":1197,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221911+0000","last_change":"2024-09-16T13:28:28.979247+0000","last_active":"2024-09-16T13:28:58.221911+0000","last_peered":"2024-09-16T13:28:58.221911+0000","last_clean":"2024-09-16T13:28:58.221911+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:28:58.221911+0000","last_undegraded":"2024-09-16T13:28:58.221911+0000","last_fullsized":"2024-09-16T13:28:58.221911+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039287600000000001,"stat_sum":{"num_bytes":24780927,"num_objects":29,"num_object_clones":10,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":7,"num_read":1663,"num_read_kb":9891,"num_write":104,"num_write_kb":6267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"57'68","reported_seq":626,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.160193+0000","last_change":"2024-09-16T13:28:53.162614+0000","last_active":"2024-09-16T13:28:58.160193+0000","last_peered":"2024-09-16T13:28:58.160193+0000","last_clean":"2024-09-16T13:28:58.160193+0000","last_became_active":"2024-09-16T13:28:53.086651+0000","last_became_peered":"2024-09-16T13:28:53.086651+0000","last_unstale":"2024-09-16T13:28:58.160193+0000","last_undegraded":"2024-09-16T13:28:58.160193+0000","last_fullsized":"2024-09-16T13:28:58.160193+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:50.595475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080754000000000004,"stat_sum":{"num_bytes":10070264,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":8,"num_read":802,"num_read_kb":12939,"num_write":75,"num_write_kb":7602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":7625158,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"62'129","reported_seq":493,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.147068+0000","last_change":"2024-09-16T13:28:30.062793+0000","last_active":"2024-09-16T13:29:03.147068+0000","last_peered":"2024-09-16T13:29:03.147068+0000","last_clean":"2024-09-16T13:29:03.147068+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:29:03.147068+0000","last_undegraded":"2024-09-16T13:29:03.147068+0000","last_fullsized":"2024-09-16T13:29:03.147068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039817100000000001,"stat_sum":{"num_bytes":7804752,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":823,"num_read_kb":1783,"num_write":137,"num_write_kb":1885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"59'290","reported_seq":1415,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.131601+0000","last_change":"2024-09-16T13:28:55.918381+0000","last_active":"2024-09-16T13:28:59.131601+0000","last_peered":"2024-09-16T13:28:59.131601+0000","last_clean":"2024-09-16T13:28:59.131601+0000","last_became_active":"2024-09-16T13:28:55.917981+0000","last_became_peered":"2024-09-16T13:28:55.917981+0000","last_unstale":"2024-09-16T13:28:59.131601+0000","last_undegraded":"2024-09-16T13:28:59.131601+0000","last_fullsized":"2024-09-16T13:28:59.131601+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:03.006738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054,"stat_sum":{"num_bytes":62610518,"num_objects":34,"num_object_clones":8,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":13,"num_read":2246,"num_read_kb":16605,"num_write":497,"num_write_kb":8894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8196910,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"58'193","reported_seq":1431,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.218466+0000","last_change":"2024-09-16T13:28:28.984368+0000","last_active":"2024-09-16T13:28:58.218466+0000","last_peered":"2024-09-16T13:28:58.218466+0000","last_clean":"2024-09-16T13:28:58.218466+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:28:58.218466+0000","last_undegraded":"2024-09-16T13:28:58.218466+0000","last_fullsized":"2024-09-16T13:28:58.218466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000271229,"stat_sum":{"num_bytes":5405397,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":13,"num_read":2422,"num_read_kb":4871,"num_write":235,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"57'83","reported_seq":544,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.112460+0000","last_change":"2024-09-16T13:28:28.979874+0000","last_active":"2024-09-16T13:28:58.112460+0000","last_peered":"2024-09-16T13:28:58.112460+0000","last_clean":"2024-09-16T13:28:58.112460+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:28:58.112460+0000","last_undegraded":"2024-09-16T13:28:58.112460+0000","last_fullsized":"2024-09-16T13:28:58.112460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000816496,"stat_sum":{"num_bytes":34385391,"num_objects":26,"num_object_clones":9,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":10,"num_read":464,"num_read_kb":9900,"num_write":78,"num_write_kb":2149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"62'375","reported_seq":897,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.012944+0000","last_change":"2024-09-16T13:28:28.979687+0000","last_active":"2024-09-16T13:29:03.012944+0000","last_peered":"2024-09-16T13:29:03.012944+0000","last_clean":"2024-09-16T13:29:03.012944+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:29:03.012944+0000","last_undegraded":"2024-09-16T13:29:03.012944+0000","last_fullsized":"2024-09-16T13:29:03.012944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035537999999999998,"stat_sum":{"num_bytes":4587631,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":5,"num_read":2546,"num_read_kb":4636,"num_write":585,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"58'174","reported_seq":2131,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.225394+0000","last_change":"2024-09-16T13:28:55.918299+0000","last_active":"2024-09-16T13:28:58.225394+0000","last_peered":"2024-09-16T13:28:58.225394+0000","last_clean":"2024-09-16T13:28:58.225394+0000","last_became_active":"2024-09-16T13:28:55.917756+0000","last_became_peered":"2024-09-16T13:28:55.917756+0000","last_unstale":"2024-09-16T13:28:58.225394+0000","last_undegraded":"2024-09-16T13:28:58.225394+0000","last_fullsized":"2024-09-16T13:28:58.225394+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:17:58.519127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035811000000000003,"stat_sum":{"num_bytes":7410330,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":10,"num_read":2217,"num_read_kb":3736,"num_write":214,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":535741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"58'151","reported_seq":486,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.223903+0000","last_change":"2024-09-16T13:28:28.983418+0000","last_active":"2024-09-16T13:28:58.223903+0000","last_peered":"2024-09-16T13:28:58.223903+0000","last_clean":"2024-09-16T13:28:58.223903+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:28:58.223903+0000","last_undegraded":"2024-09-16T13:28:58.223903+0000","last_fullsized":"2024-09-16T13:28:58.223903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044577700000000001,"stat_sum":{"num_bytes":2371619,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":5,"num_read":1270,"num_read_kb":3214,"num_write":218,"num_write_kb":2261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.148006+0000","last_change":"2024-09-16T13:28:53.082764+0000","last_active":"2024-09-16T13:29:02.148006+0000","last_peered":"2024-09-16T13:29:02.148006+0000","last_clean":"2024-09-16T13:29:02.148006+0000","last_became_active":"2024-09-16T13:28:53.082485+0000","last_became_peered":"2024-09-16T13:28:53.082485+0000","last_unstale":"2024-09-16T13:29:02.148006+0000","last_undegraded":"2024-09-16T13:29:02.148006+0000","last_fullsized":"2024-09-16T13:29:02.148006+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:48:45.781441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"58'181","reported_seq":503,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.222397+0000","last_change":"2024-09-16T13:28:28.979658+0000","last_active":"2024-09-16T13:28:58.222397+0000","last_peered":"2024-09-16T13:28:58.222397+0000","last_clean":"2024-09-16T13:28:58.222397+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:28:58.222397+0000","last_undegraded":"2024-09-16T13:28:58.222397+0000","last_fullsized":"2024-09-16T13:28:58.222397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066082900000000004,"stat_sum":{"num_bytes":12755214,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":8,"num_read":808,"num_read_kb":11943,"num_write":253,"num_write_kb":7508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":55,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.153088+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:02.153088+0000","last_peered":"2024-09-16T13:29:02.153088+0000","last_clean":"2024-09-16T13:29:02.153088+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:29:02.153088+0000","last_undegraded":"2024-09-16T13:29:02.153088+0000","last_fullsized":"2024-09-16T13:29:02.153088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"58'143","reported_seq":906,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221855+0000","last_change":"2024-09-16T13:28:55.915810+0000","last_active":"2024-09-16T13:28:58.221855+0000","last_peered":"2024-09-16T13:28:58.221855+0000","last_clean":"2024-09-16T13:28:58.221855+0000","last_became_active":"2024-09-16T13:28:55.915373+0000","last_became_peered":"2024-09-16T13:28:55.915373+0000","last_unstale":"2024-09-16T13:28:58.221855+0000","last_undegraded":"2024-09-16T13:28:58.221855+0000","last_fullsized":"2024-09-16T13:28:58.221855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:47.104564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043631399999999998,"stat_sum":{"num_bytes":10481989,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1057,"num_read_kb":2678,"num_write":205,"num_write_kb":1397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1074310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.147957+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:29:02.147957+0000","last_peered":"2024-09-16T13:29:02.147957+0000","last_clean":"2024-09-16T13:29:02.147957+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:29:02.147957+0000","last_undegraded":"2024-09-16T13:29:02.147957+0000","last_fullsized":"2024-09-16T13:29:02.147957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"62'303","reported_seq":2104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.132909+0000","last_change":"2024-09-16T13:28:30.062555+0000","last_active":"2024-09-16T13:29:03.132909+0000","last_peered":"2024-09-16T13:29:03.132909+0000","last_clean":"2024-09-16T13:29:03.132909+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:29:03.132909+0000","last_undegraded":"2024-09-16T13:29:03.132909+0000","last_fullsized":"2024-09-16T13:29:03.132909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000230199,"stat_sum":{"num_bytes":47472852,"num_objects":39,"num_object_clones":10,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":11,"num_read":2749,"num_read_kb":41917,"num_write":453,"num_write_kb":27201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.153172+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:02.153172+0000","last_peered":"2024-09-16T13:29:02.153172+0000","last_clean":"2024-09-16T13:29:02.153172+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:29:02.153172+0000","last_undegraded":"2024-09-16T13:29:02.153172+0000","last_fullsized":"2024-09-16T13:29:02.153172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"62'60","reported_seq":228,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.123959+0000","last_change":"2024-09-16T13:28:28.979887+0000","last_active":"2024-09-16T13:29:03.123959+0000","last_peered":"2024-09-16T13:29:03.123959+0000","last_clean":"2024-09-16T13:29:03.123959+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:29:03.123959+0000","last_undegraded":"2024-09-16T13:29:03.123959+0000","last_fullsized":"2024-09-16T13:29:03.123959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053138000000000003,"stat_sum":{"num_bytes":4694136,"num_objects":22,"num_object_clones":3,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":5,"num_read":155,"num_read_kb":2216,"num_write":70,"num_write_kb":2917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.112518+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:28:58.112518+0000","last_peered":"2024-09-16T13:28:58.112518+0000","last_clean":"2024-09-16T13:28:58.112518+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:28:58.112518+0000","last_undegraded":"2024-09-16T13:28:58.112518+0000","last_fullsized":"2024-09-16T13:28:58.112518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":53,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.123775+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:28:58.123775+0000","last_peered":"2024-09-16T13:28:58.123775+0000","last_clean":"2024-09-16T13:28:58.123775+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:28:58.123775+0000","last_undegraded":"2024-09-16T13:28:58.123775+0000","last_fullsized":"2024-09-16T13:28:58.123775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"62'101","reported_seq":1764,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.143472+0000","last_change":"2024-09-16T13:28:28.979855+0000","last_active":"2024-09-16T13:29:03.143472+0000","last_peered":"2024-09-16T13:29:03.143472+0000","last_clean":"2024-09-16T13:29:03.143472+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:29:03.143472+0000","last_undegraded":"2024-09-16T13:29:03.143472+0000","last_fullsized":"2024-09-16T13:29:03.143472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039579799999999999,"stat_sum":{"num_bytes":23358961,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":1714,"num_read_kb":10101,"num_write":118,"num_write_kb":2153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.152859+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:29:02.152859+0000","last_peered":"2024-09-16T13:29:02.152859+0000","last_clean":"2024-09-16T13:29:02.152859+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:29:02.152859+0000","last_undegraded":"2024-09-16T13:29:02.152859+0000","last_fullsized":"2024-09-16T13:29:02.152859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T13:29:01.144131+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:29:01.144131+0000","last_peered":"2024-09-16T13:29:01.144131+0000","last_clean":"2024-09-16T13:29:01.144131+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:29:01.144131+0000","last_undegraded":"2024-09-16T13:29:01.144131+0000","last_fullsized":"2024-09-16T13:29:01.144131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"62'134","reported_seq":492,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.007684+0000","last_change":"2024-09-16T13:28:28.980209+0000","last_active":"2024-09-16T13:29:03.007684+0000","last_peered":"2024-09-16T13:29:03.007684+0000","last_clean":"2024-09-16T13:29:03.007684+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:29:03.007684+0000","last_undegraded":"2024-09-16T13:29:03.007684+0000","last_fullsized":"2024-09-16T13:29:03.007684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069467499999999996,"stat_sum":{"num_bytes":48269494,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":10,"num_read":373,"num_read_kb":3970,"num_write":162,"num_write_kb":3421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"53'1","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.177088+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:29:02.177088+0000","last_peered":"2024-09-16T13:29:02.177088+0000","last_clean":"2024-09-16T13:29:02.177088+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:29:02.177088+0000","last_undegraded":"2024-09-16T13:29:02.177088+0000","last_fullsized":"2024-09-16T13:29:02.177088+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.152935+0000","last_change":"2024-09-16T13:28:53.086425+0000","last_active":"2024-09-16T13:29:02.152935+0000","last_peered":"2024-09-16T13:29:02.152935+0000","last_clean":"2024-09-16T13:29:02.152935+0000","last_became_active":"2024-09-16T13:28:53.086128+0000","last_became_peered":"2024-09-16T13:28:53.086128+0000","last_unstale":"2024-09-16T13:29:02.152935+0000","last_undegraded":"2024-09-16T13:29:02.152935+0000","last_fullsized":"2024-09-16T13:29:02.152935+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:41:23.848293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"62'127","reported_seq":977,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.092762+0000","last_change":"2024-09-16T13:28:30.062973+0000","last_active":"2024-09-16T13:29:03.092762+0000","last_peered":"2024-09-16T13:29:03.092762+0000","last_clean":"2024-09-16T13:29:03.092762+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:29:03.092762+0000","last_undegraded":"2024-09-16T13:29:03.092762+0000","last_fullsized":"2024-09-16T13:29:03.092762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049121500000000003,"stat_sum":{"num_bytes":34098905,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":1736,"num_read_kb":11762,"num_write":148,"num_write_kb":13518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"53'1","reported_seq":43,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.117425+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:28:58.117425+0000","last_peered":"2024-09-16T13:28:58.117425+0000","last_clean":"2024-09-16T13:28:58.117425+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:28:58.117425+0000","last_undegraded":"2024-09-16T13:28:58.117425+0000","last_fullsized":"2024-09-16T13:28:58.117425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:20:30.803710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":52,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.123767+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:28:58.123767+0000","last_peered":"2024-09-16T13:28:58.123767+0000","last_clean":"2024-09-16T13:28:58.123767+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:28:58.123767+0000","last_undegraded":"2024-09-16T13:28:58.123767+0000","last_fullsized":"2024-09-16T13:28:58.123767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:02.177111+0000","last_change":"2024-09-16T13:28:53.093054+0000","last_active":"2024-09-16T13:29:02.177111+0000","last_peered":"2024-09-16T13:29:02.177111+0000","last_clean":"2024-09-16T13:29:02.177111+0000","last_became_active":"2024-09-16T13:28:53.083561+0000","last_became_peered":"2024-09-16T13:28:53.083561+0000","last_unstale":"2024-09-16T13:29:02.177111+0000","last_undegraded":"2024-09-16T13:29:02.177111+0000","last_fullsized":"2024-09-16T13:29:02.177111+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:47:58.719653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"62'157","reported_seq":1134,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.091406+0000","last_change":"2024-09-16T13:28:28.989279+0000","last_active":"2024-09-16T13:29:03.091406+0000","last_peered":"2024-09-16T13:29:03.091406+0000","last_clean":"2024-09-16T13:29:03.091406+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:29:03.091406+0000","last_undegraded":"2024-09-16T13:29:03.091406+0000","last_fullsized":"2024-09-16T13:29:03.091406+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024121900000000001,"stat_sum":{"num_bytes":23522467,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":6,"num_read":1478,"num_read_kb":10265,"num_write":161,"num_write_kb":4481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"58'201","reported_seq":1704,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.221673+0000","last_change":"2024-09-16T13:28:55.916522+0000","last_active":"2024-09-16T13:28:58.221673+0000","last_peered":"2024-09-16T13:28:58.221673+0000","last_clean":"2024-09-16T13:28:58.221673+0000","last_became_active":"2024-09-16T13:28:55.915965+0000","last_became_peered":"2024-09-16T13:28:55.915965+0000","last_unstale":"2024-09-16T13:28:58.221673+0000","last_undegraded":"2024-09-16T13:28:58.221673+0000","last_fullsized":"2024-09-16T13:28:58.221673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:07.234321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024448900000000002,"stat_sum":{"num_bytes":7229453,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":7,"num_read":2559,"num_read_kb":4183,"num_write":198,"num_write_kb":2882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1503462,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"58'104","reported_seq":1683,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.220187+0000","last_change":"2024-09-16T13:28:28.979223+0000","last_active":"2024-09-16T13:28:58.220187+0000","last_peered":"2024-09-16T13:28:58.220187+0000","last_clean":"2024-09-16T13:28:58.220187+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:28:58.220187+0000","last_undegraded":"2024-09-16T13:28:58.220187+0000","last_fullsized":"2024-09-16T13:28:58.220187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039311499999999998,"stat_sum":{"num_bytes":46214333,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":9,"num_read":1685,"num_read_kb":12195,"num_write":142,"num_write_kb":4831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"62'147","reported_seq":1687,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.020699+0000","last_change":"2024-09-16T13:28:28.990059+0000","last_active":"2024-09-16T13:29:03.020699+0000","last_peered":"2024-09-16T13:29:03.020699+0000","last_clean":"2024-09-16T13:29:03.020699+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:29:03.020699+0000","last_undegraded":"2024-09-16T13:29:03.020699+0000","last_fullsized":"2024-09-16T13:29:03.020699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086978299999999995,"stat_sum":{"num_bytes":29129227,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":9,"num_read":1933,"num_read_kb":27564,"num_write":194,"num_write_kb":21936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"62'89","reported_seq":2647,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.087506+0000","last_change":"2024-09-16T13:28:30.062577+0000","last_active":"2024-09-16T13:29:03.087506+0000","last_peered":"2024-09-16T13:29:03.087506+0000","last_clean":"2024-09-16T13:29:03.087506+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:29:03.087506+0000","last_undegraded":"2024-09-16T13:29:03.087506+0000","last_fullsized":"2024-09-16T13:29:03.087506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029103499999999998,"stat_sum":{"num_bytes":53936305,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":8,"num_read":2568,"num_read_kb":18429,"num_write":144,"num_write_kb":15805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"59'200","reported_seq":1914,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T13:28:59.152584+0000","last_change":"2024-09-16T13:28:28.979711+0000","last_active":"2024-09-16T13:28:59.152584+0000","last_peered":"2024-09-16T13:28:59.152584+0000","last_clean":"2024-09-16T13:28:59.152584+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:28:59.152584+0000","last_undegraded":"2024-09-16T13:28:59.152584+0000","last_fullsized":"2024-09-16T13:28:59.152584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071623100000000001,"stat_sum":{"num_bytes":31342720,"num_objects":23,"num_object_clones":3,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":3183,"num_read_kb":3374,"num_write":268,"num_write_kb":627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"58'102","reported_seq":908,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.215706+0000","last_change":"2024-09-16T13:28:28.979910+0000","last_active":"2024-09-16T13:28:58.215706+0000","last_peered":"2024-09-16T13:28:58.215706+0000","last_clean":"2024-09-16T13:28:58.215706+0000","last_became_active":"2024-09-16T13:28:26.987044+0000","last_became_peered":"2024-09-16T13:28:26.987044+0000","last_unstale":"2024-09-16T13:28:58.215706+0000","last_undegraded":"2024-09-16T13:28:58.215706+0000","last_fullsized":"2024-09-16T13:28:58.215706+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:11.809480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084175000000000005,"stat_sum":{"num_bytes":25073636,"num_objects":31,"num_object_clones":9,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":12,"num_read":808,"num_read_kb":4578,"num_write":105,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"58'90","reported_seq":289,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T13:28:58.219884+0000","last_change":"2024-09-16T13:28:28.980116+0000","last_active":"2024-09-16T13:28:58.219884+0000","last_peered":"2024-09-16T13:28:58.219884+0000","last_clean":"2024-09-16T13:28:58.219884+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:28:58.219884+0000","last_undegraded":"2024-09-16T13:28:58.219884+0000","last_fullsized":"2024-09-16T13:28:58.219884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010142790000000001,"stat_sum":{"num_bytes":13207063,"num_objects":20,"num_object_clones":6,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":11,"num_read":188,"num_read_kb":5675,"num_write":83,"num_write_kb":3773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'443","reported_seq":1748,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T13:29:03.076389+0000","last_change":"2024-09-16T13:28:58.119992+0000","last_active":"2024-09-16T13:29:03.076389+0000","last_peered":"2024-09-16T13:29:03.076389+0000","last_clean":"2024-09-16T13:29:03.076389+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:29:03.076389+0000","last_undegraded":"2024-09-16T13:29:03.076389+0000","last_fullsized":"2024-09-16T13:29:03.076389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074187000000000005,"stat_sum":{"num_bytes":24966524,"num_objects":34,"num_object_clones":11,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":14,"num_read":2971,"num_read_kb":5720,"num_write":701,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":769272799,"num_objects":795,"num_object_clones":197,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":260,"num_read":55748,"num_read_kb":289170,"num_write":6538,"num_write_kb":169049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":22372624,"num_keys_recovered":36,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34095104,"data_stored":348647513,"data_compressed":7760012,"data_compressed_allocated":26775552,"data_compressed_original":344188258,"omap_allocated":0,"internal_metadata":0},"log_size":4990,"ondisk_log_size":4990,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":17628,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":0,"seq":64424509455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306472,"kb_used_data":4956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94065368,"statfs":{"total":96636764160,"available":96322936832,"internally_reserved":0,"allocated":5074944,"data_stored":54199431,"data_compressed":904982,"data_compressed_allocated":3571712,"data_compressed_original":53354303,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305868,"kb_used_data":4348,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94065972,"statfs":{"total":96636764160,"available":96323555328,"internally_reserved":0,"allocated":4452352,"data_stored":29397079,"data_compressed":1206553,"data_compressed_allocated":2756608,"data_compressed_original":28443506,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310648,"kb_used_data":4976,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061192,"statfs":{"total":96636764160,"available":96318660608,"internally_reserved":0,"allocated":5095424,"data_stored":32708194,"data_compressed":1155483,"data_compressed_allocated":2981888,"data_compressed_original":31551488,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":6,"up_from":14,"seq":60129542162,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":5092,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":5214208,"data_stored":49988340,"data_compressed":1048267,"data_compressed_allocated":3739648,"data_compressed_original":49242994,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303744,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068096,"statfs":{"total":96636764160,"available":96325730304,"internally_reserved":0,"allocated":2277376,"data_stored":10930704,"data_compressed":446790,"data_compressed_allocated":1052672,"data_compressed_original":10317824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.3390000000000004,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":9.4480000000000004,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314388,"kb_used_data":8716,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94057452,"statfs":{"total":96636764160,"available":96314830848,"internally_reserved":0,"allocated":8925184,"data_stored":89037783,"data_compressed":1665396,"data_compressed_allocated":6668288,"data_compressed_original":87748608,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307572,"kb_used_data":6056,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94064268,"statfs":{"total":96636764160,"available":96321810432,"internally_reserved":0,"allocated":6201344,"data_stored":71128756,"data_compressed":942129,"data_compressed_allocated":4685824,"data_compressed_original":70238015,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5435392,"data_stored":70223462,"data_compressed":937009,"data_compressed_allocated":4661248,"data_compressed_original":69689151,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8220672,"data_stored":88719646,"data_compressed":1665480,"data_compressed_allocated":6672384,"data_compressed_original":87756800,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1662976,"data_stored":11205379,"data_compressed":448652,"data_compressed_allocated":1150976,"data_compressed_original":10924032,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4337664,"data_stored":32332766,"data_compressed":1155483,"data_compressed_allocated":2981888,"data_compressed_original":31551488,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3788800,"data_stored":29091552,"data_compressed":1206553,"data_compressed_allocated":2756608,"data_compressed_original":28443506,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":53893885,"data_compressed":904982,"data_compressed_allocated":3571712,"data_compressed_original":53354303,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":49987047,"data_compressed":1055108,"data_compressed_allocated":3756032,"data_compressed_original":49546098,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":13193776,"data_compressed":386745,"data_compressed_allocated":1224704,"data_compressed_original":12922880,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:29:04.878 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T13:29:04.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:04.878 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-16T13:29:04.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-09-16T13:29:04.951 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:04.952 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:04.952 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:04.952 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:04.952 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:04.952 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:04.950+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:04.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:05.043 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:05.042+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.935555034, 2024-09-16T13:29:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.930079468999999, 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.169752662, 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.153214537, 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8145 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4456448~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810633968000001, 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041847961000000003, 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8145 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290586+0000", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290587+0000", 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.03e-07 2024-09-16T13:29:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290589+0000", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.086e-06 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290596+0000", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1509999999999998e-06 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290600+0000", 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.692e-06 2024-09-16T13:29:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332033+0000", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041433316999999997 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332120+0000", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6847000000000004e-05 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332434+0000", 2024-09-16T13:29:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031446499999999999 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8146 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5373952~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810617008000001, 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042182996, 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8146 2024-09-16T13:29:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290603+0000", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290604+0000", 2024-09-16T13:29:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290605+0000", 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290612+0000", 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.686e-06 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290616+0000", 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332462+0000", 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041846176999999998 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332534+0000", 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2398999999999996e-05 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332786+0000", 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025215300000000002 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8147 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810600936, 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042474660999999997, 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8147 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290619+0000", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290620+0000", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-07 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290621+0000", 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290627+0000", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9020000000000004e-06 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290632+0000", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6609999999999999e-06 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332812+0000", 2024-09-16T13:29:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042180278000000002 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.332854+0000", 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1857999999999998e-05 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333094+0000", 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000239905 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810584655, 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810567677000002, 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:29:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810551087, 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810528741999999, 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810510762, 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810494958, 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810479397000002, 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810461683, 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.810435099999999, 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.968881187000001, 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.771441616000001, 2024-09-16T13:29:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.760942869000001, 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:29:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:29:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.1965650099999996, 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.198 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:05.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:05.200 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:29:05.200 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:05.197+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:05.205 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T13:29:05.204+0000 7fb7555f1640 -1 mon.a@0(leader).osd e65 definitely_dead 0 2024-09-16T13:29:05.232 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone18 from 0xebfffc (overlap 0xebfffc) to 0x2f3332 2024-09-16T13:29:05.301 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:05.299+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:05.334 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:05.334 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:05.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51133 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [call rbd.sparse_copyup in=1048616b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.018564710000000002, 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018585609999999999, 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51133 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.335 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313991+0000", 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4323e-05 2024-09-16T13:29:05.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314605+0000", 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061453400000000002 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314617+0000", 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1199000000000001e-05 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314622+0000", 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2360000000000004e-06 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314665+0000", 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3058e-05 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314816+0000", 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000150784 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315045+0000", 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022899899999999999 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325487+0000", 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010442475999999999 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.339 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51134 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [truncate 622592] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.017907854000000001, 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017975569, 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51134 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314634+0000", 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8700000000000005e-07 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314637+0000", 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7180000000000001e-06 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314644+0000", 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8210000000000004e-06 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314649+0000", 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2299999999999999e-06 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324223+0000", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095740400000000007 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324325+0000", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101879 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324488+0000", 2024-09-16T13:29:05.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016357700000000001 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325748+0000", 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001259451 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51139 4.1c e65/32 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head v 65'112, mlcod=65'110)", 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321744+0000", 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.010796750000000001, 2024-09-16T13:29:05.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010907729, 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321744+0000", 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321744+0000", 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321745+0000", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8899999999999998e-07 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321982+0000", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000236352 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321995+0000", 2024-09-16T13:29:05.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3399e-05 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.322002+0000", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324202+0000", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022006310000000002 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324221+0000", 2024-09-16T13:29:05.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8563000000000002e-05 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51140 4.1c e65/32)", 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.329155+0000", 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.0033864189999999999, 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003549398, 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "queued for pg", 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329155+0000", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329155+0000", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329156+0000", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1230000000000001e-06 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329163+0000", 2024-09-16T13:29:05.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3699999999999997e-06 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329176+0000", 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2497000000000001e-05 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329182+0000", 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3080000000000004e-06 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:51133 4.4 e65/32)", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.330019+0000", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.0025220239999999999, 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027287100000000001, 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "queued for pg", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330019+0000", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330019+0000", 2024-09-16T13:29:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330020+0000", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8700000000000005e-07 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330024+0000", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4349999999999999e-06 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330037+0000", 2024-09-16T13:29:05.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2529e-05 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330042+0000", 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.354 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:51134 4.4 e65/32)", 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.330059+0000", 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.0024825559999999999, 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027370290000000002, 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "queued for pg", 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330059+0000", 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330059+0000", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330059+0000", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.15e-07 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330061+0000", 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9649999999999998e-06 2024-09-16T13:29:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330066+0000", 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3140000000000003e-06 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330069+0000", 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.446e-06 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.358 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:05.358 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 6 2024-09-16T13:29:05.358 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:05.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:05.370 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-09-16T13:29:05.414 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:05.412+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:05.490 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:05.490 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:05.490 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:05.490 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:05.490 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:05.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:05.501 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:05.499+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:05.603 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:05.601+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:05.638 INFO:teuthology.orchestra.run.smithi076.stdout:2308 trunc from 0xebfffc to 0x8dc7052 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.198734809000001, 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:29:05.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:29:05.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:29:05.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.026335632999999, 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.791263759, 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:29:05.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:29:05.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:29:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.938563637000001, 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:05.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:05.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:05.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:05.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.917499728999999, 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.669 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:05.673 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.613426691000001, 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.674 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:29:05.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.508832868999999, 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:29:05.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:29:05.684 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.502289227999999, 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:05.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:29:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23819 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.825388167, 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018047839999999999, 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.689 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23819 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815691+0000", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.054e-06 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815696+0000", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1749999999999996e-06 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815708+0000", 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1446e-05 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815715+0000", 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824348+0000", 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086330599999999997 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824416+0000", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8213999999999994e-05 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824679+0000", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026342200000000003 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826830+0000", 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002150749 2024-09-16T13:29:05.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833601+0000", 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067705100000000004 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833698+0000", 2024-09-16T13:29:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7188000000000005e-05 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833738+0000", 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0173000000000003e-05 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23820 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.825357513, 2024-09-16T13:29:05.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018255289000000001, 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23820 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815725+0000", 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7840000000000001e-06 2024-09-16T13:29:05.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815732+0000", 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9040000000000003e-06 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815737+0000", 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5549999999999996e-06 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825011+0000", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092747530000000002 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825098+0000", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6287999999999998e-05 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825475+0000", 2024-09-16T13:29:05.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037692200000000002 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826843+0000", 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001368403 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833748+0000", 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069050989999999996 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833884+0000", 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013585 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833976+0000", 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2219999999999995e-05 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23821 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.825336119999999, 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018354471000000001, 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23821 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815743+0000", 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.46e-07 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815747+0000", 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889999999999999e-06 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815756+0000", 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-06 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815760+0000", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825875+0000", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010114911000000001 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825919+0000", 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3572000000000002e-05 2024-09-16T13:29:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826099+0000", 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018034099999999999 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827140+0000", 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010402899999999999 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833988+0000", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068483349999999997 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834065+0000", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7305999999999995e-05 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834097+0000", 2024-09-16T13:29:05.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1492000000000003e-05 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.010529469000002, 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:05.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:05.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:05.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.001652890999999, 2024-09-16T13:29:05.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:29:05.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:29:05.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.841658680999998, 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:05.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:29:05.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.841625652000001, 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:05.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:29:05.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:05.734 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.830261757999999, 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:28298 4.7 4:e7bc7339:::rbd_data.11a9320547a2.0000000000000003:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.341022537000001, 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033840752000000002, 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 28298 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.739 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300057+0000", 2024-09-16T13:29:05.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300062+0000", 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2820000000000004e-06 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300075+0000", 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2975e-05 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300081+0000", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9830000000000001e-06 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300108+0000", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7155999999999999e-05 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300156+0000", 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7311000000000002e-05 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.302362+0000", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002206253 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333731+0000", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031369085999999997 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333794+0000", 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2712999999999994e-05 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333852+0000", 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8013999999999997e-05 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333897+0000", 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4728999999999998e-05 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:28298 4.7 e51/32 ondisk, result = 0)", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.332614413, 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025439566, 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308465+0000", 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308472+0000", 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0480000000000001e-06 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308494+0000", 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1712000000000002e-05 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308504+0000", 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4509999999999998e-06 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333763+0000", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025259193999999999 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333785+0000", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2497000000000002e-05 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333904+0000", 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118263 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.748 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51133 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [call rbd.sparse_copyup in=1048616b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.32710192700000001, 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019127457, 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51133 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313991+0000", 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4323e-05 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314605+0000", 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061453400000000002 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314617+0000", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1199000000000001e-05 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314622+0000", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2360000000000004e-06 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314665+0000", 2024-09-16T13:29:05.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3058e-05 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314816+0000", 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000150784 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315045+0000", 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022899899999999999 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325487+0000", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010442475999999999 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333017+0000", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007529488 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333065+0000", 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8878000000000002e-05 2024-09-16T13:29:05.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333104+0000", 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8482000000000002e-05 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.755 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51134 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [truncate 622592] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.32644507099999998, 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018568774999999999, 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51134 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314634+0000", 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8700000000000005e-07 2024-09-16T13:29:05.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314637+0000", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7180000000000001e-06 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314644+0000", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8210000000000004e-06 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314649+0000", 2024-09-16T13:29:05.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2299999999999999e-06 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324223+0000", 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095740400000000007 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324325+0000", 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101879 2024-09-16T13:29:05.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324488+0000", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016357700000000001 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325748+0000", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001259451 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333137+0000", 2024-09-16T13:29:05.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073892690000000004 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333170+0000", 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2997000000000003e-05 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:05.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333202+0000", 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2005999999999998e-05 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:05.762 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:05.763 INFO:teuthology.orchestra.run.smithi076.stdout:2309 write 0x8873ac0 thru 0x88757c1 (0x1d02 bytes) 2024-09-16T13:29:05.763 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:05.763 INFO:teuthology.orchestra.run.smithi076.stdout:2313 trunc from 0x8dc7052 to 0xaacc6d6 2024-09-16T13:29:05.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:05.767 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:05.767 INFO:teuthology.orchestra.run.smithi076.stdout:2315 punch from 0x982098e to 0x982b5ed, (0xac5f bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2317 read 0x6b2f2e8 thru 0x6b3bb0c (0xc825 bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2318 read 0x8a5cab9 thru 0x8a5d409 (0x951 bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2319 write 0x3152265 thru 0x3155cfe (0x3a9a bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2321 read 0x1c6dd78 thru 0x1c7c0e6 (0xe36f bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2322 read 0x1cfd6d2 thru 0x1cffc47 (0x2576 bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2323 trunc from 0xaacc6d6 to 0xc9bddcd 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2324 writesame 0x4661a03 thru 0x4664262 data_size 0x11(0x2860 bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2326 read 0x680e64f thru 0x680ec62 (0x614 bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2327 read 0x3946cbb thru 0x3951b24 (0xae6a bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2330 read 0x6131f69 thru 0x6135e43 (0x3edb bytes) 2024-09-16T13:29:05.768 INFO:teuthology.orchestra.run.smithi076.stdout:2331 read 0x7f9c1ac thru 0x7fa1e49 (0x5c9e bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2332 read 0x6b9dea2 thru 0x6ba0af2 (0x2c51 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2333 writesame 0x9abafb8 thru 0x9abd4ab data_size 0x37(0x24f4 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2334 read 0x3054f33 thru 0x30598b7 (0x4985 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2336 write 0xc18adc5 thru 0xc18c9b6 (0x1bf2 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2338 write 0xba9c52d thru 0xbaa929d (0xcd71 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2339 trunc from 0xc9bddcd to 0x8ac5e57 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2340 write 0xebec595 thru 0xebf6609 (0xa075 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2341 writesame 0xc2c26ce thru 0xc2c821c data_size 0x19(0x5b4f bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2344 write 0x35b8d05 thru 0x35c129d (0x8599 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2345 read 0xa0b227 thru 0xa11a6c (0x6846 bytes) 2024-09-16T13:29:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:2348 trunc from 0xebf660a to 0xd739922 2024-09-16T13:29:05.770 INFO:teuthology.orchestra.run.smithi076.stdout:2351 read 0xd95648 thru 0xd9dd40 (0x86f9 bytes) 2024-09-16T13:29:05.770 INFO:teuthology.orchestra.run.smithi076.stdout:2352 writesame 0x5688462 thru 0x5691f33 data_size 0x26(0x9ad2 bytes) 2024-09-16T13:29:05.770 INFO:teuthology.orchestra.run.smithi076.stdout:2353 read 0x747c57a thru 0x7488942 (0xc3c9 bytes) 2024-09-16T13:29:05.770 INFO:teuthology.orchestra.run.smithi076.stdout:2355 clone 20 order 19 su 8192 sc 14 2024-09-16T13:29:05.770 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:05.762+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:05.864 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:05.863+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:05.904 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:05.904 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:05.904 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:05.904 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:05.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:05.997 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:05.995+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:06.051 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:06.051 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:06.051 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:06.051 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:06.051 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:06.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:06.066 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:06.064+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:06.167 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:06.165+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:06.181 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.7. 2024-09-16T13:29:06.198 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1933 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.129808869999998, 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088826889999999992, 2024-09-16T13:29:06.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1933 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077514+0000", 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077526+0000", 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1834000000000001e-05 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077529+0000", 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.76e-06 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083235+0000", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057052270000000002 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083314+0000", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9680999999999998e-05 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083493+0000", 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017889199999999999 2024-09-16T13:29:06.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084872+0000", 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013790250000000001 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086316+0000", 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014437289999999999 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086358+0000", 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1962999999999998e-05 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086393+0000", 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5657000000000001e-05 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2319 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.090597635999998, 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080751409999999992, 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2319 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116724+0000", 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1619999999999998e-06 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116732+0000", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.554e-06 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116751+0000", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9454999999999999e-05 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116758+0000", 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0650000000000001e-06 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116832+0000", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4058000000000001e-05 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116965+0000", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013282999999999999 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124797+0000", 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078320170000000001 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.220 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.075601267000003, 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:06.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:06.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:39596 4.4 e61/32 4:21c2e4aa:::rbd_data.11a9a07602f8.0000000000000000:head v 61'94, mlcod=58'92)", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.6306599070000001, 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084428249999999993, 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576661+0000", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.209e-06 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577410+0000", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074892800000000005 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577428+0000", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8566000000000001e-05 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577435+0000", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8789999999999997e-06 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577489+0000", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4079000000000002e-05 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577515+0000", 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5690999999999998e-05 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.585055+0000", 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075399940000000004 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.585103+0000", 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7478999999999997e-05 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.684783353, 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:06.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:06.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:06.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:06.236 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.047108884, 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:06.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:06.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:06.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50058 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 688128~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.242 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.237493+0000", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96982709, 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080396170000000006, 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50058 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237493+0000", 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237493+0000", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237493+0000", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237495+0000", 2024-09-16T13:29:06.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.891e-06 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237501+0000", 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5440000000000004e-06 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.237505+0000", 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3500000000000001e-06 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.245388+0000", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078830919999999995 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.245452+0000", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4165000000000007e-05 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.245532+0000", 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0190000000000003e-05 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.246 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50244 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96720540899999996, 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080470409999999996, 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50244 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240115+0000", 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240129+0000", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4602e-05 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240136+0000", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4509999999999997e-06 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240139+0000", 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.247852+0000", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077129629999999998 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.247922+0000", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9558e-05 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248161+0000", 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023963899999999999 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.250 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50245 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96717597799999999, 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081528850000000003, 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50245 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.251 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1600000000000002e-07 2024-09-16T13:29:06.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240146+0000", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0109999999999999e-06 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240153+0000", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.092e-06 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240157+0000", 2024-09-16T13:29:06.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4589999999999998e-06 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248188+0000", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080313069999999997 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248235+0000", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7462000000000001e-05 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248297+0000", 2024-09-16T13:29:06.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1137999999999996e-05 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50246 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 581632~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96715934699999995, 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083633849999999992, 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.255 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50246 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240161+0000", 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240162+0000", 2024-09-16T13:29:06.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6729999999999999e-06 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240168+0000", 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6550000000000001e-06 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240179+0000", 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0740000000000001e-05 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248340+0000", 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081612739999999996 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248420+0000", 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9856000000000003e-05 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248524+0000", 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010387600000000001 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50247 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96713672900000003, 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008526034, 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50247 2024-09-16T13:29:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240185+0000", 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0710000000000002e-06 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240192+0000", 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9099999999999999e-06 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240196+0000", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5700000000000001e-06 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248544+0000", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083479139999999997 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248616+0000", 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2713999999999995e-05 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248709+0000", 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2477000000000002e-05 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50248 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96712006900000003, 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086742339999999994, 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.264 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50248 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9799999999999999e-07 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240207+0000", 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240213+0000", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.353e-06 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240217+0000", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3349999999999998e-06 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248762+0000", 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085456879999999992 2024-09-16T13:29:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248816+0000", 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3297000000000001e-05 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248874+0000", 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8192999999999997e-05 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50249 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96709940100000003, 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088346970000000007, 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50249 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240221+0000", 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1199999999999999e-07 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240222+0000", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5349999999999999e-06 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240228+0000", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5999999999999997e-06 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240231+0000", 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.286e-06 2024-09-16T13:29:06.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248916+0000", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086852720000000008 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248989+0000", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3030999999999999e-05 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249055+0000", 2024-09-16T13:29:06.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5660999999999994e-05 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50250 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96708532999999997, 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089558599999999995, 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50250 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240235+0000", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.41e-07 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240236+0000", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.469e-06 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240242+0000", 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5589999999999996e-06 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240245+0000", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0199999999999999e-06 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249091+0000", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088466229999999996 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249133+0000", 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1457e-05 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249190+0000", 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7391e-05 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50251 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96707159099999995, 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091258229999999999, 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.277 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50251 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240250+0000", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4500000000000001e-06 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240255+0000", 2024-09-16T13:29:06.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6459999999999998e-06 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240259+0000", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1889999999999999e-06 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249224+0000", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089657999999999995 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249275+0000", 2024-09-16T13:29:06.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0504000000000003e-05 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249374+0000", 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8917000000000002e-05 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96705767899999995, 2024-09-16T13:29:06.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:06.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.96703035900000001, 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:06.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:06.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.89207684700000001, 2024-09-16T13:29:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:06.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88678157000000002, 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:06.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:06.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:06.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:06.300 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.88622301199999998, 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:06.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:06.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:06.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:06.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.306 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:06.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:06.308 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone19 from 0xd739922 (overlap 0xebfffc) to 0x5e6664 2024-09-16T13:29:06.308 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:06.306+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:06.409 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:06.407+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:06.447 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:06.447 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.202185589, 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.202205674, 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:06.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:06.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20209171300000001, 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20215070600000001, 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.452 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.202063724, 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20215502099999999, 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.456 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:06.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:06.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20078969799999999, 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20091213599999999, 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:06.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20076645200000001, 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20092290300000001, 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:06.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:06.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.467 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20361331899999999, 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20380139799999999, 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:06.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:06.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:06.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20355084600000001, 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.203769804, 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:06.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:06.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:06.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.203506193, 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20375707400000001, 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:06.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20343309200000001, 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20371486999999999, 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.479 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:06.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20341458600000001, 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20372720899999999, 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.483 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:06.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.203240634, 2024-09-16T13:29:06.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.203583978, 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:06.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:06.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.20320208000000001, 2024-09-16T13:29:06.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.20357647100000001, 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:06.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:06.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4.3a92eb60 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.202243386, 2024-09-16T13:29:06.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.202648417, 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:06.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:06.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 13 2024-09-16T13:29:06.499 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:06.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:06.510 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:06.508+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:06.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:06.599 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:06.599 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:06.600 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:06.600 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:06.600 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:06.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:06.611 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:06.609+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:06.705 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:06.705 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:06.711 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:06.710+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:06.721 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:06.721 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:06.735 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:06.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:06.750 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:06.751 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:06.751 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:06.751 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:06.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:06.767 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:06.768 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:06.784 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:06.784 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:06.800 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:06.800 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:06.813 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:06.811+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:06.817 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:06.817 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:06.874 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:06.875 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:06.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:06.925 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:06.923+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:07.015 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:07.013+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:07.025 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.025 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:07.118 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:07.116+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:07.174 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.174 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:07.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:29:07.226 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:07.224+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:07.317 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:07.315+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:07.326 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.326 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.335 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:07.417 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:07.416+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:07.472 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.473 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:07.526 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:07.524+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:07.579 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.579 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:07.629 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:07.627+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:07.720 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:07.719+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:07.726 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.726 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:07.821 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:07.820+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:07.875 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:07.875 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:07.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:07.922 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:07.921+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:08.023 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.023 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.023 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:08.021+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:08.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:08.124 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:08.122+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:08.132 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.132 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:08.180 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:29:08.226 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:08.224+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:08.280 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.280 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:08.326 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:08.324+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:08.427 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:08.425+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:08.528 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:08.527+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:08.542 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.542 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:08.629 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:08.627+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:08.648 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.648 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.657 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:08.730 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:08.728+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:08.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:08.848 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:08.846+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:08.904 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:08.904 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:08.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:08.957 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:08.955+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:08.965 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:08.980 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:08.981 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:08.995 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:08.996 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:09.011 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:09.014 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.014 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:09.014 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:09.015 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:09.015 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:09.027 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:09.027 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:09.032 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:09.030+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:09.050 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:09.050 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:09.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:09.081 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:09.103 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:09.104 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:09.133 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:09.131+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:09.164 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-09-16T13:29:09.165 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.165 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:09.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:09.165 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:09.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1146 4.f 4:f5740b61:::rbd_data.11a962dad130.00000000000001b1:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.719797106999998, 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017005851999999998, 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1146 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442344+0000", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442345+0000", 2024-09-16T13:29:09.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442352+0000", 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442366+0000", 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442371+0000", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2499999999999997e-06 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442411+0000", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9563000000000001e-05 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.442520+0000", 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109028 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.443372+0000", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00085211599999999998 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459238+0000", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015866203999999998 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459282+0000", 2024-09-16T13:29:09.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3958000000000002e-05 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459321+0000", 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8849e-05 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:33.459349+0000", 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8289e-05 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:3132 4.9 e38/32 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head v 38'24, mlcod=38'21)", 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.547397930999999, 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017534220999999999, 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614743+0000", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.614744+0000", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3599999999999999e-06 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617703+0000", 2024-09-16T13:29:09.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029586090000000001 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617725+0000", 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617732+0000", 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3919999999999997e-06 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617783+0000", 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.617805+0000", 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2365e-05 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632235+0000", 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014429343000000001 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.632277+0000", 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2311999999999997e-05 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.312326057, 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.176 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:09.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:29:09.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:29:09.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:29:09.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.459625934999998, 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.182 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:09.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.438562027, 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:09.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:09.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.134488989000001, 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:29:09.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:29:09.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:09.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21649 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'42, mlcod=46'40)", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:09.197 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.029895166999999, 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017326036, 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:09.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132246+0000", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3600000000000003e-07 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132552+0000", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030615500000000001 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132564+0000", 2024-09-16T13:29:09.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132569+0000", 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3059999999999997e-06 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132607+0000", 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7926999999999998e-05 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.132626+0000", 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9264999999999999e-05 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149520+0000", 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016893679000000002 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.149572+0000", 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:21651 4.10 e46/32 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head v 46'44, mlcod=46'40)", 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.023351525999999, 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015550151, 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138789+0000", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138790+0000", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5099999999999999e-07 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138796+0000", 2024-09-16T13:29:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0759999999999999e-06 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139287+0000", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00049084900000000002 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.139293+0000", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147243+0000", 2024-09-16T13:29:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079502300000000008 2024-09-16T13:29:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.147263+0000", 2024-09-16T13:29:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9626000000000001e-05 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154207+0000", 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069443810000000003 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154339+0000", 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013195399999999999 2024-09-16T13:29:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23819 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.346450465, 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018047839999999999, 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23819 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815690+0000", 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815691+0000", 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.054e-06 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815696+0000", 2024-09-16T13:29:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1749999999999996e-06 2024-09-16T13:29:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815708+0000", 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1446e-05 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815715+0000", 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.849e-06 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824348+0000", 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086330599999999997 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824416+0000", 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8213999999999994e-05 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.824679+0000", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026342200000000003 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826830+0000", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002150749 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833601+0000", 2024-09-16T13:29:09.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067705100000000004 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833698+0000", 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7188000000000005e-05 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833738+0000", 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0173000000000003e-05 2024-09-16T13:29:09.215 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23820 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.346419811000001, 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018255289000000001, 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23820 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815721+0000", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815725+0000", 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7840000000000001e-06 2024-09-16T13:29:09.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815732+0000", 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9040000000000003e-06 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815737+0000", 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5549999999999996e-06 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825011+0000", 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092747530000000002 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825098+0000", 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6287999999999998e-05 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825475+0000", 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037692200000000002 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826843+0000", 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001368403 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833748+0000", 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069050989999999996 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833884+0000", 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013585 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833976+0000", 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2219999999999995e-05 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:23821 4.12 4:4a3d74b3:::rbd_object_map.11a9ff22d86c:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.346398418, 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018354471000000001, 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.223 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 23821 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815742+0000", 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815743+0000", 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.46e-07 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815747+0000", 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2889999999999999e-06 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815756+0000", 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-06 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.815760+0000", 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825875+0000", 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010114911000000001 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.825919+0000", 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3572000000000002e-05 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.826099+0000", 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018034099999999999 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.827140+0000", 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010402899999999999 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.833988+0000", 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068483349999999997 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834065+0000", 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7305999999999995e-05 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.834097+0000", 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1492000000000003e-05 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.230 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.531591766999998, 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:09.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:09.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.522715188999999, 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:09.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.242 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.362720978999999, 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:29:09.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.362687950000002, 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.250 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:29:09.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:09.256 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.351324055999999, 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:09.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:29:09.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:28298 4.7 4:e7bc7339:::rbd_data.11a9320547a2.0000000000000003:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:09.261 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.862084835000001, 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033840752000000002, 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 28298 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:09.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300057+0000", 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300062+0000", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2820000000000004e-06 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300075+0000", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2975e-05 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300081+0000", 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9830000000000001e-06 2024-09-16T13:29:09.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300108+0000", 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7155999999999999e-05 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300156+0000", 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7311000000000002e-05 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.302362+0000", 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002206253 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333731+0000", 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031369085999999997 2024-09-16T13:29:09.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333794+0000", 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2712999999999994e-05 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333852+0000", 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8013999999999997e-05 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333897+0000", 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4728999999999998e-05 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:28298 4.7 e51/32 ondisk, result = 0)", 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:09.268 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.853676710999999, 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025439566, 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308465+0000", 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308472+0000", 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0480000000000001e-06 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308494+0000", 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1712000000000002e-05 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308504+0000", 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4509999999999998e-06 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333763+0000", 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025259193999999999 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333785+0000", 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2497000000000002e-05 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333904+0000", 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118263 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51133 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [call rbd.sparse_copyup in=1048616b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.8481642250000001, 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019127457, 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.274 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51133 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:09.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313991+0000", 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4323e-05 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314605+0000", 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061453400000000002 2024-09-16T13:29:09.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314617+0000", 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1199000000000001e-05 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314622+0000", 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2360000000000004e-06 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314665+0000", 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3058e-05 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314816+0000", 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000150784 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315045+0000", 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022899899999999999 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325487+0000", 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010442475999999999 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333017+0000", 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007529488 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333065+0000", 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8878000000000002e-05 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333104+0000", 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8482000000000002e-05 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51134 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [truncate 622592] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.8475073690000001, 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018568774999999999, 2024-09-16T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51134 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:09.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314633+0000", 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314634+0000", 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8700000000000005e-07 2024-09-16T13:29:09.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314637+0000", 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7180000000000001e-06 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314644+0000", 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8210000000000004e-06 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314649+0000", 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2299999999999999e-06 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324223+0000", 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095740400000000007 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324325+0000", 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101879 2024-09-16T13:29:09.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.324488+0000", 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016357700000000001 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325748+0000", 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001259451 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333137+0000", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073892690000000004 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333170+0000", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2997000000000003e-05 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333202+0000", 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2005999999999998e-05 2024-09-16T13:29:09.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.288 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.288 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.288 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:2356 trunc from 0xd739922 to 0x3b14c4c 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:2358 writesame 0xdc29adb thru 0xdc342d9 data_size 0x1(0xa7ff bytes) 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:2359 read 0x7c26a55 thru 0x7c2a89d (0x3e49 bytes) 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:2360 punch from 0x7ca0169 to 0x7cacd4a, (0xcbe1 bytes) 2024-09-16T13:29:09.291 INFO:teuthology.orchestra.run.smithi076.stdout:2361 write 0xcb3a09 thru 0xcb5b94 (0x218c bytes) 2024-09-16T13:29:09.292 INFO:teuthology.orchestra.run.smithi076.stdout:2362 read 0x4162f47 thru 0x416dd10 (0xadca bytes) 2024-09-16T13:29:09.292 INFO:teuthology.orchestra.run.smithi076.stdout:2364 write 0xdac8f56 thru 0xdad13e8 (0x8493 bytes) 2024-09-16T13:29:09.292 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:09.287+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:09.390 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:09.388+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.392 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:09.392 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:09.491 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:09.490+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:09.539 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.539 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:09.539 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:09.539 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:09.539 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:09.592 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:09.590+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:09.646 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.646 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:1933 4.1c 4:3d73c564:::rbd_data.11a962dad130.000000000000001c:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.566274587999999, 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088826889999999992, 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1933 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077511+0000", 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2799999999999996e-07 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077514+0000", 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077526+0000", 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1834000000000001e-05 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.077529+0000", 2024-09-16T13:29:09.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.76e-06 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083235+0000", 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057052270000000002 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083314+0000", 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9680999999999998e-05 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.083493+0000", 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017889199999999999 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.084872+0000", 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013790250000000001 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086316+0000", 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014437289999999999 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086358+0000", 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1962999999999998e-05 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.086393+0000", 2024-09-16T13:29:09.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5657000000000001e-05 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2319 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [stat,delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.527063353999999, 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080751409999999992, 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.654 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2319 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116722+0000", 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116724+0000", 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1619999999999998e-06 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116732+0000", 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.554e-06 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116751+0000", 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9454999999999999e-05 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116758+0000", 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0650000000000001e-06 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116832+0000", 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4058000000000001e-05 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.116965+0000", 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013282999999999999 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.124797+0000", 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078320170000000001 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.512066984999997, 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:09.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:09.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:09.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:39596 4.4 e61/32 4:21c2e4aa:::rbd_data.11a9a07602f8.0000000000000000:head v 61'94, mlcod=58'92)", 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.0671256249999992, 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084428249999999993, 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576660+0000", 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.576661+0000", 2024-09-16T13:29:09.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.209e-06 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577410+0000", 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074892800000000005 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577428+0000", 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8566000000000001e-05 2024-09-16T13:29:09.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577435+0000", 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8789999999999997e-06 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577489+0000", 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4079000000000002e-05 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.577515+0000", 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5690999999999998e-05 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.585055+0000", 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075399940000000004 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:01.585103+0000", 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7478999999999997e-05 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.1212490710000003, 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.672 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:09.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:09.678 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.483574602, 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:09.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:09.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:09.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50244 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.403671127, 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080470409999999996, 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50244 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240114+0000", 2024-09-16T13:29:09.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240115+0000", 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240129+0000", 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4602e-05 2024-09-16T13:29:09.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240136+0000", 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4509999999999997e-06 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240139+0000", 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.247852+0000", 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077129629999999998 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.247922+0000", 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9558e-05 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248161+0000", 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023963899999999999 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50245 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4036416960000002, 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081528850000000003, 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.692 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50245 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240144+0000", 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1600000000000002e-07 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240146+0000", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0109999999999999e-06 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240153+0000", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.092e-06 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240157+0000", 2024-09-16T13:29:09.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4589999999999998e-06 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248188+0000", 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080313069999999997 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248235+0000", 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7462000000000001e-05 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248297+0000", 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1137999999999996e-05 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50246 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 581632~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:09.697 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4036250649999999, 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083633849999999992, 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50246 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240160+0000", 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240161+0000", 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1100000000000002e-07 2024-09-16T13:29:09.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240162+0000", 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6729999999999999e-06 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240168+0000", 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6550000000000001e-06 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240179+0000", 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0740000000000001e-05 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248340+0000", 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0081612739999999996 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248420+0000", 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9856000000000003e-05 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248524+0000", 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010387600000000001 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.702 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50247 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 598016~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4036024469999999, 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008526034, 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50247 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240183+0000", 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240185+0000", 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0710000000000002e-06 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240192+0000", 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9099999999999999e-06 2024-09-16T13:29:09.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240196+0000", 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5700000000000001e-06 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248544+0000", 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083479139999999997 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248616+0000", 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2713999999999995e-05 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248709+0000", 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2477000000000002e-05 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50248 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4035857869999999, 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086742339999999994, 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50248 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240200+0000", 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9799999999999999e-07 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240207+0000", 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:29:09.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240213+0000", 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.353e-06 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240217+0000", 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3349999999999998e-06 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248762+0000", 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085456879999999992 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248816+0000", 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3297000000000001e-05 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248874+0000", 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8192999999999997e-05 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50249 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4035651189999996, 2024-09-16T13:29:09.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088346970000000007, 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50249 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:09.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240220+0000", 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240221+0000", 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1199999999999999e-07 2024-09-16T13:29:09.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240222+0000", 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5349999999999999e-06 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240228+0000", 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5999999999999997e-06 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240231+0000", 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.286e-06 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248916+0000", 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086852720000000008 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.248989+0000", 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3030999999999999e-05 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249055+0000", 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5660999999999994e-05 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50250 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4035510479999997, 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089558599999999995, 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50250 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:09.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240235+0000", 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.41e-07 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240236+0000", 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.469e-06 2024-09-16T13:29:09.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240242+0000", 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5589999999999996e-06 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240245+0000", 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0199999999999999e-06 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249091+0000", 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088466229999999996 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249133+0000", 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1457e-05 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249190+0000", 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7391e-05 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50251 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4035373089999998, 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091258229999999999, 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50251 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240250+0000", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4500000000000001e-06 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240255+0000", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6459999999999998e-06 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240259+0000", 2024-09-16T13:29:09.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1889999999999999e-06 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249224+0000", 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089657999999999995 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249275+0000", 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0504000000000003e-05 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249374+0000", 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8917000000000002e-05 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4035233969999998, 2024-09-16T13:29:09.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:09.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:09.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.4034960769999998, 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:09.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:09.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:09.739 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3285425650000002, 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3232472880000001, 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.745 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:09.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:09.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:09.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.3226887300000003, 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.752 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:09.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:09.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52548 4.1c 4:3c81e3d1:::rbd_data.11a9fc41ce3c.0000000000000073:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 229376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.201714+0000", 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.442071508, 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082721570000000005, 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:09.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52548 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201714+0000", 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:09.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201714+0000", 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201716+0000", 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0049999999999999e-06 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201725+0000", 2024-09-16T13:29:09.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0089999999999996e-06 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201745+0000", 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0627000000000001e-05 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201753+0000", 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8080000000000005e-06 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201811+0000", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8115999999999997e-05 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.201818+0000", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3520000000000003e-06 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.206010+0000", 2024-09-16T13:29:09.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041926180000000004 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.206120+0000", 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109454 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.206265+0000", 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014531400000000001 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.207294+0000", 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010291969999999999 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.209929+0000", 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026344229999999999 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.209962+0000", 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3568000000000001e-05 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.209986+0000", 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3666e-05 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:09.767 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:09.769 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:09.766+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:09.868 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:09.867+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:09.909 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:09.909 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:09.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:09.923 INFO:teuthology.orchestra.run.smithi076.stdout:2365 write 0xc81c9c7 thru 0xc8252aa (0x88e4 bytes) 2024-09-16T13:29:09.934 INFO:teuthology.orchestra.run.smithi076.stdout:2366 writesame 0x748ce63 thru 0x7492667 data_size 0x57(0x5805 bytes) 2024-09-16T13:29:09.943 INFO:teuthology.orchestra.run.smithi076.stdout:2367 read 0x648c0c5 thru 0x648da93 (0x19cf bytes) 2024-09-16T13:29:09.943 INFO:teuthology.orchestra.run.smithi076.stdout:2368 write 0x67c0959 thru 0x67c5b67 (0x520f bytes) 2024-09-16T13:29:09.951 INFO:teuthology.orchestra.run.smithi076.stdout:2369 write 0x97b4057 thru 0x97bdfed (0x9f97 bytes) 2024-09-16T13:29:09.961 INFO:teuthology.orchestra.run.smithi076.stdout:2370 read 0xbfe654a thru 0xbfeac67 (0x471e bytes) 2024-09-16T13:29:09.968 INFO:teuthology.orchestra.run.smithi076.stdout:2371 writesame 0xec4621a thru 0xec4a9b3 data_size 0x6(0x479a bytes) 2024-09-16T13:29:09.968 INFO:teuthology.orchestra.run.smithi076.stdout:2372 write 0x35806c5 thru 0x3584385 (0x3cc1 bytes) 2024-09-16T13:29:09.969 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:09.967+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:10.024 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:10.025 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:10.025 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:10.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:10.074 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:10.072+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:10.170 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:10.169+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:10.173 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:3297 4.17 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2069521~10445] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.503566061000001, 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030440354999999999, 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.174 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3297 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:10.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667258+0000", 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668628+0000", 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001369501 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668651+0000", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3017999999999999e-05 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668658+0000", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668704+0000", 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5479000000000003e-05 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668825+0000", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121791 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.669018+0000", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019308300000000001 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.685320+0000", 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016301690000000001 2024-09-16T13:29:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697626+0000", 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012306375 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697668+0000", 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999997e-05 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697698+0000", 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0006999999999999e-05 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8556 4.0 4:017e3810:::rbd_data.11a91a08d9ca.00000000000000d0:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.230268457000001, 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03408597, 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8556 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940556+0000", 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.127e-06 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940560+0000", 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940570+0000", 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302000000000001e-05 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940574+0000", 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940608+0000", 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3748000000000003e-05 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940660+0000", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.942061+0000", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014009750000000001 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974517+0000", 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032456609999999997 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974529+0000", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2235000000000001e-05 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974593+0000", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974641+0000", 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7459000000000001e-05 2024-09-16T13:29:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8556 4.0 e42/32 ondisk, result = 0)", 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.224144319000001, 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027841225000000001, 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946680+0000", 2024-09-16T13:29:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946683+0000", 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6009999999999999e-06 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946694+0000", 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0949e-05 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946699+0000", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7740000000000004e-06 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974498+0000", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027798386000000001 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974509+0000", 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1157e-05 2024-09-16T13:29:10.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974520+0000", 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1517e-05 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21649 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.040796840999999, 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020572536999999998, 2024-09-16T13:29:10.190 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21649 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130027+0000", 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130028+0000", 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3510000000000001e-06 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130517+0000", 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00048930600000000001 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130532+0000", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4752e-05 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130538+0000", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1689999999999997e-06 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131149+0000", 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061055899999999999 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131241+0000", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1978000000000006e-05 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.131807+0000", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056649900000000004 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141368+0000", 2024-09-16T13:29:10.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095605849999999999 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150523+0000", 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009155046 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150564+0000", 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649000000000001e-05 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.150599+0000", 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4643e-05 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21651 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.040245183, 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024369312000000001, 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21651 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130579+0000", 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9999999999999998e-07 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130581+0000", 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.537e-06 2024-09-16T13:29:10.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130587+0000", 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130591+0000", 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138200+0000", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007608325 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138265+0000", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5007999999999998e-05 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138437+0000", 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017266100000000001 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141654+0000", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032171859999999999 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154881+0000", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013226236000000001 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154915+0000", 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4471999999999998e-05 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154947+0000", 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2372000000000001e-05 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49432 4.10 4:09c05623:::rbd_data.11a9b5ab4b7f.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6190015940000002, 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025173402000000001, 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49432 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551823+0000", 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.376e-06 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551830+0000", 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8140000000000003e-06 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551843+0000", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2665e-05 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551848+0000", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2059999999999998e-06 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551886+0000", 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8642e-05 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551976+0000", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9247999999999997e-05 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.553110+0000", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001134471 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576835+0000", 2024-09-16T13:29:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023724463000000001 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576884+0000", 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9734999999999997e-05 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576951+0000", 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6120999999999999e-05 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576995+0000", 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4660999999999998e-05 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:49432 4.10 e64/32 ondisk, result = 0)", 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6145576830000001, 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020739172, 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.208 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:10.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3200000000000005e-07 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556269+0000", 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0510000000000001e-06 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556279+0000", 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4760000000000004e-06 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556283+0000", 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576867+0000", 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020583946999999998 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576876+0000", 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6360000000000006e-06 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.577005+0000", 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012836999999999999 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9285747989999997, 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.212 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:10.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:10.218 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9285123259999999, 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:10.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:10.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9284676730000001, 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:10.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:10.233 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9283945720000002, 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:10.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9283760660000002, 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.239 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.244 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9282021139999999, 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:10.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:10.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:29:10.250 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9281635599999998, 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9272048659999998, 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:10.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:10.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:10.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.262 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9271470690000001, 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:10.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:29:10.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9270531929999999, 2024-09-16T13:29:10.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:10.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:10.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.927025204, 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.274 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:10.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:10.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.9257511780000001, 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:10.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.925727932, 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:29:10.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:10.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:10.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:10.292 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:10.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:29:10.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:10.295 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:10.292+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:10.394 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:10.395 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:10.395 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:10.395 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:10.395 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:10.393+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:10.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:10.496 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:10.494+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:10.500 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:10.501 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:10.501 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:10.501 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:10.501 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:10.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:10.597 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:10.595+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:10.650 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:10.650 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:10.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:10.650 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3132 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 4368762~22099] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.039228727999998, 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024228047999999999, 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3132 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608311+0000", 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4300000000000002e-07 2024-09-16T13:29:10.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614081+0000", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057699559999999997 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614094+0000", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2672e-05 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614100+0000", 2024-09-16T13:29:10.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.071e-06 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614155+0000", 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614248+0000", 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2958e-05 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:10.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614457+0000", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020964000000000001 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.629775+0000", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015317084 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632471+0000", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026965330000000001 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632510+0000", 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8831000000000001e-05 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632538+0000", 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8330000000000002e-05 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7024 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:10.656 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.264498322000001, 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015178159, 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7024 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:10.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1899999999999998e-07 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383042+0000", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.57e-06 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383047+0000", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-06 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383050+0000", 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398103+0000", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052963000000001 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398151+0000", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8863999999999998e-05 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398219+0000", 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7291000000000003e-05 2024-09-16T13:29:10.660 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.937200101000002, 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:10.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.666 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.92766099, 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:10.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.670 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25661 4.d 4:b5278be0:::rbd_data.11a926bfcc53.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.016959812, 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022042392000000001, 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25661 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:10.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630580+0000", 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.37e-06 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630587+0000", 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.477e-06 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630597+0000", 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0473e-05 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630619+0000", 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2050000000000001e-05 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630649+0000", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9513999999999999e-05 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630704+0000", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4753999999999998e-05 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.631810+0000", 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011062489999999999 2024-09-16T13:29:10.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652526+0000", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020715588 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652559+0000", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3859999999999998e-05 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652594+0000", 2024-09-16T13:29:10.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4749000000000002e-05 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652621+0000", 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7308e-05 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25661 4.d e48/32 ondisk, result = 0)", 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.012378007999999, 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017464550999999998, 2024-09-16T13:29:10.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635162+0000", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3200000000000004e-07 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635165+0000", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5449999999999999e-06 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635178+0000", 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2461e-05 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635182+0000", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.403e-06 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652548+0000", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017365642000000001 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652555+0000", 2024-09-16T13:29:10.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652625+0000", 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0648000000000005e-05 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.961204254999998, 2024-09-16T13:29:10.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:10.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:10.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.952154406999998, 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.686 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:29:10.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.250792804, 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:10.690 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:10.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:10.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.240145755, 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.696 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:10.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:10.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:10.699 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.573084174, 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.700 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.705 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38134 4.1b e59/32 ondisk, result = 0)", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.563287816000001, 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036326266000000003, 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084252+0000", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1039999999999999e-06 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084256+0000", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5820000000000002e-06 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084270+0000", 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3752e-05 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084276+0000", 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6449999999999999e-06 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120563+0000", 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036286410999999998 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120571+0000", 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2139999999999996e-06 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120577+0000", 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.709 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50477 4.1 4:824a833b:::rbd_data.11a9b5ab4b7f.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.37582249, 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016452580000000001, 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50477 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271718+0000", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4440000000000001e-06 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272533+0000", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081477100000000005 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272547+0000", 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4780000000000001e-05 2024-09-16T13:29:10.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272554+0000", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4389999999999997e-06 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272600+0000", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6511999999999998e-05 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272671+0000", 2024-09-16T13:29:10.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0494000000000006e-05 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.273230+0000", 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055887199999999997 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288080+0000", 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014850070999999999 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288086+0000", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2400000000000004e-06 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288123+0000", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7302999999999998e-05 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288169+0000", 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5654e-05 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50478 4.1 4:824a833b:::rbd_data.11a9b5ab4b7f.0000000000000005:head [delete] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.3749710879999997, 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015740127, 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.716 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50478 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272569+0000", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4199999999999995e-07 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272571+0000", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8430000000000001e-06 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272578+0000", 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8870000000000003e-06 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272583+0000", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2479999999999998e-06 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279404+0000", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068219409999999998 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279508+0000", 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010376499999999999 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279706+0000", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000198032 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288205+0000", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084983509999999995 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288219+0000", 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4511e-05 2024-09-16T13:29:10.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288274+0000", 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4731000000000002e-05 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288308+0000", 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4075999999999997e-05 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51816 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 737280~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4023490699999996, 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015237298, 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51816 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.48e-07 2024-09-16T13:29:10.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245191+0000", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2619999999999999e-06 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245196+0000", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4059999999999998e-06 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245198+0000", 2024-09-16T13:29:10.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2970000000000002e-06 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260130+0000", 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014932124999999999 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260176+0000", 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6211999999999997e-05 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260427+0000", 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025074800000000002 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51817 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 327680~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.402338759, 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015346831, 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.726 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51817 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245201+0000", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245202+0000", 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.28e-06 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245205+0000", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1149999999999998e-06 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245207+0000", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.17e-06 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260443+0000", 2024-09-16T13:29:10.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015236275000000001 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260490+0000", 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6252999999999999e-05 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260547+0000", 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7296000000000002e-05 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51818 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4023298129999997, 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015474253, 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51818 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245211+0000", 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.646e-06 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245217+0000", 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4370000000000004e-06 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245221+0000", 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260574+0000", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015352852 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260623+0000", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9394999999999997e-05 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260683+0000", 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0071000000000001e-05 2024-09-16T13:29:10.734 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51819 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 540672~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4023141289999996, 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015585564999999999, 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.735 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51819 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245227+0000", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1610000000000001e-06 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245234+0000", 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6189999999999996e-06 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245238+0000", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0300000000000004e-06 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260713+0000", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015474913 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260754+0000", 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1063999999999999e-05 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260810+0000", 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6388999999999998e-05 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51820 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 999424~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.402296915, 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015898123, 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.739 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51820 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245244+0000", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.883e-06 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245250+0000", 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7470000000000002e-06 2024-09-16T13:29:10.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245254+0000", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.596e-06 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260837+0000", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015583052999999999 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260890+0000", 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3776999999999997e-05 2024-09-16T13:29:10.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261140+0000", 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000249683 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51821 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 589824~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.4022820500000002, 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016004555, 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:10.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51821 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1600000000000002e-07 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245259+0000", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245275+0000", 2024-09-16T13:29:10.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5767999999999999e-05 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245284+0000", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261159+0000", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015874632999999999 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261205+0000", 2024-09-16T13:29:10.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6817999999999998e-05 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261261+0000", 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6039999999999999e-05 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.747 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:10.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:10.749 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:10.747+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:10.849 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:10.848+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4.534fc9e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:10.888 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.91196825199999998, 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.911990733, 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "waiting for peered", 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:10.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:10.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:10.892 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:29:10.893 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:29:10.893 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:10.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:10.916 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:29:10.922 INFO:teuthology.orchestra.run.smithi076.stdout:2373 writesame 0x9216b82 thru 0x922161c data_size 0x5(0xaa9b bytes) 2024-09-16T13:29:10.935 INFO:teuthology.orchestra.run.smithi076.stdout:2374 writesame 0xd0f3add thru 0xd0fa6da data_size 0x17(0x6bfe bytes) 2024-09-16T13:29:10.944 INFO:teuthology.orchestra.run.smithi076.stdout:2375 trunc from 0xec4a9b4 to 0xd3410bb 2024-09-16T13:29:10.950 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:10.948+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:10.953 INFO:teuthology.orchestra.run.smithi076.stdout:2376 write 0x3b89b4b thru 0x3b8f632 (0x5ae8 bytes) 2024-09-16T13:29:10.955 INFO:teuthology.orchestra.run.smithi076.stdout:2378 punch from 0x49b09ba to 0x49b3e8e, (0x34d4 bytes) 2024-09-16T13:29:10.955 INFO:teuthology.orchestra.run.smithi076.stdout:2380 write 0x2aeb386 thru 0x2af10ed (0x5d68 bytes) 2024-09-16T13:29:10.962 INFO:teuthology.orchestra.run.smithi076.stdout:2381 write 0xb5176ad thru 0xb51f17f (0x7ad3 bytes) 2024-09-16T13:29:10.970 INFO:teuthology.orchestra.run.smithi076.stdout:2382 write 0x68ba3dd thru 0x68c6894 (0xc4b8 bytes) 2024-09-16T13:29:10.979 INFO:teuthology.orchestra.run.smithi076.stdout:2384 read 0x95a7615 thru 0x95b36b8 (0xc0a4 bytes) 2024-09-16T13:29:10.980 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:10.980 INFO:teuthology.orchestra.run.smithi076.stdout:2387 read 0xa7b6ba5 thru 0xa7ba652 (0x3aae bytes) 2024-09-16T13:29:10.980 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:10.980 INFO:teuthology.orchestra.run.smithi076.stdout:2389 write 0x343e4ce thru 0x3447802 (0x9335 bytes) 2024-09-16T13:29:10.991 INFO:teuthology.orchestra.run.smithi076.stdout:2393 write 0xa7112 thru 0xb3948 (0xc837 bytes) 2024-09-16T13:29:11.020 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:11.021 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:11.021 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:11.021 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:11.021 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:11.021 INFO:teuthology.orchestra.run.smithi076.stdout:2394 writesame 0x36b291c thru 0x36b9f82 data_size 0x11(0x7667 bytes) 2024-09-16T13:29:11.025 INFO:teuthology.orchestra.run.smithi076.stdout:2396 write 0xa6f3c57 thru 0xa6fd35f (0x9709 bytes) 2024-09-16T13:29:11.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:11.038 INFO:teuthology.orchestra.run.smithi076.stdout:2398 write 0x76e6647 thru 0x76ec991 (0x634b bytes) 2024-09-16T13:29:11.047 INFO:teuthology.orchestra.run.smithi076.stdout:2400 read 0xac9eecb thru 0xacac1f4 (0xd32a bytes) 2024-09-16T13:29:11.047 INFO:teuthology.orchestra.run.smithi076.stdout:2401 writesame 0xcf7ba1f thru 0xcf85ed8 data_size 0x5(0xa4ba bytes) 2024-09-16T13:29:11.057 INFO:teuthology.orchestra.run.smithi076.stdout:2402 writesame 0x994fb36 thru 0x995c9f3 data_size 0x3(0xcebe bytes) 2024-09-16T13:29:11.068 INFO:teuthology.orchestra.run.smithi076.stdout:2403 write 0xb728ea7 thru 0xb731d56 (0x8eb0 bytes) 2024-09-16T13:29:11.073 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:11.071+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:11.078 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.079 INFO:teuthology.orchestra.run.smithi076.stdout:2405 write 0x3b6c12e thru 0x3b7bd45 (0xfc18 bytes) 2024-09-16T13:29:11.091 INFO:teuthology.orchestra.run.smithi076.stdout:2406 trunc from 0xd3410bb to 0x5f19000 2024-09-16T13:29:11.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:2407 writesame 0x730a698 thru 0x730bd44 data_size 0x2b(0x16ad bytes) 2024-09-16T13:29:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:2408 write 0x5362764 thru 0x5368a63 (0x6300 bytes) 2024-09-16T13:29:11.123 INFO:teuthology.orchestra.run.smithi076.stdout:2409 punch from 0x5022955 to 0x5026325, (0x39d0 bytes) 2024-09-16T13:29:11.123 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:2411 write 0x8319c4c thru 0x831b3a4 (0x1759 bytes) 2024-09-16T13:29:11.134 INFO:teuthology.orchestra.run.smithi076.stdout:2412 punch from 0x23aa0c5 to 0x23aaead, (0xde8 bytes) 2024-09-16T13:29:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:2413 write 0xe2f0651 thru 0xe2f45d6 (0x3f86 bytes) 2024-09-16T13:29:11.152 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:11.150+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:11.152 INFO:teuthology.orchestra.run.smithi076.stdout:2414 write 0xb0092a thru 0xb0b060 (0xa737 bytes) 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.183 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.541127905000003, 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:11.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:11.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:11.189 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.450597500000001, 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:11.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:11.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:11.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:11.194 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.423468329999999, 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:11.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.198 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.007706177999999, 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:11.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:11.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.937369896, 2024-09-16T13:29:11.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:11.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:11.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.494248150000001, 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:11.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:11.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.213 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.484158882999999, 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.217 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 20.337370248999999, 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:11.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:11.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50161 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 565248~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9320744310000002, 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019286728999999999, 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50161 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9499999999999998e-07 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248542+0000", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248547+0000", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4870000000000002e-06 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248551+0000", 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6500000000000002e-06 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267735+0000", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019183892000000001 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267783+0000", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7840999999999998e-05 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267826+0000", 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3695000000000001e-05 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50162 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 581632~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.932059508, 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019455525000000001, 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50162 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.228 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248555+0000", 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:11.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248557+0000", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7850000000000001e-06 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248562+0000", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8579999999999996e-06 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248566+0000", 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.207e-06 2024-09-16T13:29:11.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267854+0000", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019288254000000001 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267938+0000", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3678999999999996e-05 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268010+0000", 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2416000000000006e-05 2024-09-16T13:29:11.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50163 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 598016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9320446570000005, 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019564818000000001, 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.232 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50163 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248570+0000", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248577+0000", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4320000000000002e-06 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248584+0000", 2024-09-16T13:29:11.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248587+0000", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3790000000000002e-06 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268040+0000", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019453427999999998 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268096+0000", 2024-09-16T13:29:11.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5404e-05 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268134+0000", 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50164 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 614400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9320229820000003, 2024-09-16T13:29:11.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019695615, 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50164 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248593+0000", 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9300000000000002e-06 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248599+0000", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7950000000000004e-06 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248603+0000", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268171+0000", 2024-09-16T13:29:11.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019568232000000001 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268243+0000", 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1872000000000006e-05 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268287+0000", 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3300000000000002e-05 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50165 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 630784~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9320067940000003, 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019831332, 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50165 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.241 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248608+0000", 2024-09-16T13:29:11.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248610+0000", 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.807e-06 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248619+0000", 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4779999999999997e-06 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248623+0000", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0489999999999997e-06 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268313+0000", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019690151999999999 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268379+0000", 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5792999999999998e-05 2024-09-16T13:29:11.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268439+0000", 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537000000000003e-05 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50166 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9319870229999996, 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019843890999999999, 2024-09-16T13:29:11.245 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50166 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.65e-07 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248629+0000", 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.854e-06 2024-09-16T13:29:11.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248635+0000", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248645+0000", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7599999999999997e-06 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268466+0000", 2024-09-16T13:29:11.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019821442000000002 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268471+0000", 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.775e-06 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50167 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9319653930000005, 2024-09-16T13:29:11.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019857698999999999, 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50167 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248651+0000", 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.181e-06 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248657+0000", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0939999999999996e-06 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248661+0000", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268499+0000", 2024-09-16T13:29:11.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019837751000000001 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268506+0000", 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7440000000000007e-06 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50168 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:11.253 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.9319496799999998, 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019875693999999999, 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50168 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:11.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248665+0000", 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248667+0000", 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.866e-06 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248672+0000", 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3410000000000002e-06 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248675+0000", 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.264e-06 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268536+0000", 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019860932000000001 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268540+0000", 2024-09-16T13:29:11.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50928 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 311296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.883950821, 2024-09-16T13:29:11.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019281187000000002, 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50928 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0600000000000001e-07 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296665+0000", 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5069999999999999e-06 2024-09-16T13:29:11.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296670+0000", 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9930000000000003e-06 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296673+0000", 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.151e-06 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315875+0000", 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019202 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315910+0000", 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5036000000000001e-05 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315944+0000", 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4193999999999998e-05 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.264 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50929 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 327680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.8839376950000002, 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019357040999999998, 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50929 2024-09-16T13:29:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296677+0000", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9900000000000002e-07 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296678+0000", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.401e-06 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296683+0000", 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296686+0000", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8949999999999998e-06 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315967+0000", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019280701000000001 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316000+0000", 2024-09-16T13:29:11.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3549e-05 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316033+0000", 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3272999999999998e-05 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50930 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 344064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:11.269 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.8839253520000003, 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019423292000000002, 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50930 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:11.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5499999999999999e-07 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296690+0000", 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.575e-06 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296696+0000", 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3460000000000001e-06 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296704+0000", 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6319999999999997e-06 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316046+0000", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019341649999999998 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316079+0000", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2898999999999998e-05 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316112+0000", 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2935000000000002e-05 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.273 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.2054458560000001, 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:11.274 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:11.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:11.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:11.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:11.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:11.280 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:11.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:11.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:11.281 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T13:29:11.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:29:11.282 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:11.282 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:11.283 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:11.283 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:11.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:11.284 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:11.281+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:11.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:11.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:11.285 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:11.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:11.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:11.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:11.318 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:11.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:11.348 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:11.349 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:11.383 INFO:teuthology.orchestra.run.smithi076.stdout:2415 punch from 0x1f7b29f to 0x1f8aca9, (0xfa0a bytes) 2024-09-16T13:29:11.383 INFO:teuthology.orchestra.run.smithi076.stdout:2416 read 0x65a2e46 thru 0x65a5001 (0x21bc bytes) 2024-09-16T13:29:11.383 INFO:teuthology.orchestra.run.smithi076.stdout:2417 read 0x304a46e thru 0x305370c (0x929f bytes) 2024-09-16T13:29:11.384 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:11.382+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:11.384 INFO:teuthology.orchestra.run.smithi076.stdout:2420 punch from 0xba09283 to 0xba15758, (0xc4d5 bytes) 2024-09-16T13:29:11.384 INFO:teuthology.orchestra.run.smithi076.stdout:2421 punch from 0x2f12910 to 0x2f139b6, (0x10a6 bytes) 2024-09-16T13:29:11.384 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:2425 read 0x425447b thru 0x4257891 (0x3417 bytes) 2024-09-16T13:29:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:2426 write 0x1023d65 thru 0x102e1b4 (0xa450 bytes) 2024-09-16T13:29:11.399 INFO:teuthology.orchestra.run.smithi076.stdout:2427 punch from 0x86fb4ca to 0x8705dba, (0xa8f0 bytes) 2024-09-16T13:29:11.399 INFO:teuthology.orchestra.run.smithi076.stdout:2429 read 0x88419a thru 0x88f8ad (0xb714 bytes) 2024-09-16T13:29:11.400 INFO:teuthology.orchestra.run.smithi076.stdout:2430 read 0x1b098f5 thru 0x1b15ac8 (0xc1d4 bytes) 2024-09-16T13:29:11.400 INFO:teuthology.orchestra.run.smithi076.stdout:2431 write 0xb28b9ab thru 0xb29882a (0xce80 bytes) 2024-09-16T13:29:11.411 INFO:teuthology.orchestra.run.smithi076.stdout:2432 writesame 0xd209e26 thru 0xd219c47 data_size 0x15(0xfe22 bytes) 2024-09-16T13:29:11.426 INFO:teuthology.orchestra.run.smithi076.stdout:2433 read 0xa65ae76 thru 0xa6622d6 (0x7461 bytes) 2024-09-16T13:29:11.426 INFO:teuthology.orchestra.run.smithi076.stdout:2435 read 0x17ed3a thru 0x18e512 (0xf7d9 bytes) 2024-09-16T13:29:11.427 INFO:teuthology.orchestra.run.smithi076.stdout:2436 writesame 0x748bec3 thru 0x748f73c data_size 0x1(0x387a bytes) 2024-09-16T13:29:11.439 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:11.439 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:11.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:11.457 INFO:teuthology.orchestra.run.smithi076.stdout:2437 read 0x3f641c0 thru 0x3f699d1 (0x5812 bytes) 2024-09-16T13:29:11.457 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.457 INFO:teuthology.orchestra.run.smithi076.stdout:2439 read 0xd4322c7 thru 0xd435153 (0x2e8d bytes) 2024-09-16T13:29:11.457 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:11.457 INFO:teuthology.orchestra.run.smithi076.stdout:2441 write 0x62970e8 thru 0x62a443d (0xd356 bytes) 2024-09-16T13:29:11.477 INFO:teuthology.orchestra.run.smithi076.stdout:2442 trunc from 0xe2f45d7 to 0x66cd8cc 2024-09-16T13:29:11.484 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:11.482+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:11.548 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:11.549 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:11.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:11.578 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:11.578 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 71, 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:29:10.910189+0000", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:29:06.165279+0000", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.579 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 4, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:29:11.580 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:29:11.581 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:11.582 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:11.583 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:11.584 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.585 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:11.586 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:11.587 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:11.588 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:11.589 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:11.590 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:29:11.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:29:11.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:11.593 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:29:11.594 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:29:11.595 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:11.596 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "pool_client.0", 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:25.583589+0000", 2024-09-16T13:29:11.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2024-09-16T13:29:11.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "66", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:11.599 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 23, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 66, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:11.600 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:11.601 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:11.603 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 69, 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:29:11.605 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:29:11.607 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 56, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.609 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.610 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.611 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.612 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:29:11.613 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.614 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.6766510009765625, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 57, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:29:11.616 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:29:11.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:29:11.619 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.620 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.622 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.623 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.631 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:29:11.632 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.633 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:29:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:29:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:11.653 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.5", 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T13:29:11.654 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:11.655 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [], 2024-09-16T13:29:11.656 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:11.657 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:11.658 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:11.658 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:11.659 INFO:tasks.thrashosds.thrasher:j is {'epoch': 71, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:29:10.910189+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:29:06.165279+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:28:25.583589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 66, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6766510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 57, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [6, 1]}], 'pg_upmap_items': [{'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:29:11.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:29:11.660 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:11.657+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:11.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:11.697 INFO:teuthology.orchestra.run.smithi076.stdout:2443 read 0x18eba02 thru 0x18f6043 (0xa642 bytes) 2024-09-16T13:29:11.707 INFO:teuthology.orchestra.run.smithi076.stdout:2446 trunc from 0x66cd8cc to 0xe3a4764 2024-09-16T13:29:11.712 INFO:teuthology.orchestra.run.smithi076.stdout:2447 clone 21 order 22 su 1048576 sc 8 2024-09-16T13:29:11.759 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:11.757+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:11.802 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:11.802 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:11.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:11.860 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:11.858+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:11.909 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:11.909 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:11.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:11.962 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:11.959+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:11.964 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:29:11.973 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-16T13:29:11.849478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":905657173,"num_objects":974,"num_object_clones":222,"num_object_copies":1948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":304,"num_read":67575,"num_read_kb":376998,"num_write":7913,"num_write_kb":218667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":34847427,"num_keys_recovered":48,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5903,"ondisk_log_size":5903,"up":90,"acting":85,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2487396,"kb_used_data":46128,"kb_used_omap":13,"kb_used_meta":2441138,"kb_avail":752487324,"statfs":{"total":773094113280,"available":770547019776,"internally_reserved":0,"allocated":47235072,"data_stored":429722345,"data_compressed":8871394,"data_compressed_allocated":32948224,"data_compressed_original":421695454,"omap_allocated":13632,"internal_metadata":2499726016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,13],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26963653,"num_objects":103,"num_object_clones":21,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":40,"num_read":5343,"num_read_kb":61400,"num_write":446,"num_write_kb":20521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11884435,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000589"},"pg_stats":[{"pgid":"4.1f","version":"65'144","reported_seq":1041,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.183263+0000","last_change":"2024-09-16T13:28:28.989963+0000","last_active":"2024-09-16T13:29:06.183263+0000","last_peered":"2024-09-16T13:29:06.183263+0000","last_clean":"2024-09-16T13:29:06.183263+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:29:06.183263+0000","last_undegraded":"2024-09-16T13:29:06.183263+0000","last_fullsized":"2024-09-16T13:29:06.183263+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089281099999999997,"stat_sum":{"num_bytes":48516228,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":6,"num_read":1867,"num_read_kb":7501,"num_write":153,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"63'231","reported_seq":9528,"reported_epoch":67,"state":"remapped+peering","last_fresh":"2024-09-16T13:29:07.181095+0000","last_change":"2024-09-16T13:29:07.181265+0000","last_active":"2024-09-16T13:29:06.179129+0000","last_peered":"2024-09-16T13:29:05.430757+0000","last_clean":"2024-09-16T13:29:05.430757+0000","last_became_active":"2024-09-16T13:28:55.915295+0000","last_became_peered":"2024-09-16T13:28:55.915295+0000","last_unstale":"2024-09-16T13:29:07.181095+0000","last_undegraded":"2024-09-16T13:29:07.181095+0000","last_fullsized":"2024-09-16T13:29:07.181095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:30.240090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041021900000000002,"stat_sum":{"num_bytes":53235989,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":6,"num_read":10372,"num_read_kb":17794,"num_write":366,"num_write_kb":3512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1345764,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"68'168","reported_seq":459,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.045393+0000","last_change":"2024-09-16T13:28:28.984841+0000","last_active":"2024-09-16T13:29:09.045393+0000","last_peered":"2024-09-16T13:29:09.045393+0000","last_clean":"2024-09-16T13:29:09.045393+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:29:09.045393+0000","last_undegraded":"2024-09-16T13:29:09.045393+0000","last_fullsized":"2024-09-16T13:29:09.045393+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071631099999999997,"stat_sum":{"num_bytes":14994244,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":799,"num_read_kb":3580,"num_write":209,"num_write_kb":2802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"66'129","reported_seq":976,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.238864+0000","last_change":"2024-09-16T13:28:28.981596+0000","last_active":"2024-09-16T13:29:06.238864+0000","last_peered":"2024-09-16T13:29:06.238864+0000","last_clean":"2024-09-16T13:29:06.238864+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:29:06.238864+0000","last_undegraded":"2024-09-16T13:29:06.238864+0000","last_fullsized":"2024-09-16T13:29:06.238864+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030921799999999999,"stat_sum":{"num_bytes":9422346,"num_objects":29,"num_object_clones":6,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":6,"num_read":1323,"num_read_kb":11469,"num_write":141,"num_write_kb":5180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"68'99","reported_seq":426,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.091410+0000","last_change":"2024-09-16T13:28:28.979438+0000","last_active":"2024-09-16T13:29:09.091410+0000","last_peered":"2024-09-16T13:29:09.091410+0000","last_clean":"2024-09-16T13:29:09.091410+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:29:09.091410+0000","last_undegraded":"2024-09-16T13:29:09.091410+0000","last_fullsized":"2024-09-16T13:29:09.091410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056234500000000001,"stat_sum":{"num_bytes":15067716,"num_objects":29,"num_object_clones":8,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":8,"num_read":336,"num_read_kb":4406,"num_write":105,"num_write_kb":4037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"68'123","reported_seq":1357,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.145474+0000","last_change":"2024-09-16T13:28:28.979457+0000","last_active":"2024-09-16T13:29:09.145474+0000","last_peered":"2024-09-16T13:29:09.145474+0000","last_clean":"2024-09-16T13:29:09.145474+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:29:09.145474+0000","last_undegraded":"2024-09-16T13:29:09.145474+0000","last_fullsized":"2024-09-16T13:29:09.145474+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058420599999999998,"stat_sum":{"num_bytes":23772865,"num_objects":29,"num_object_clones":4,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":5,"num_read":1331,"num_read_kb":6742,"num_write":140,"num_write_kb":2067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"68'108","reported_seq":942,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.084472+0000","last_change":"2024-09-16T13:28:28.983849+0000","last_active":"2024-09-16T13:29:09.084472+0000","last_peered":"2024-09-16T13:29:09.084472+0000","last_clean":"2024-09-16T13:29:09.084472+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:29:09.084472+0000","last_undegraded":"2024-09-16T13:29:09.084472+0000","last_fullsized":"2024-09-16T13:29:09.084472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041077799999999999,"stat_sum":{"num_bytes":31028971,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":14,"num_read":850,"num_read_kb":2352,"num_write":91,"num_write_kb":1965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'145","reported_seq":1596,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.121323+0000","last_change":"2024-09-16T13:28:28.979247+0000","last_active":"2024-09-16T13:29:09.121323+0000","last_peered":"2024-09-16T13:29:09.121323+0000","last_clean":"2024-09-16T13:29:09.121323+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:29:09.121323+0000","last_undegraded":"2024-09-16T13:29:09.121323+0000","last_fullsized":"2024-09-16T13:29:09.121323+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039287600000000001,"stat_sum":{"num_bytes":50649704,"num_objects":37,"num_object_clones":12,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":8,"num_read":2033,"num_read_kb":14121,"num_write":180,"num_write_kb":9185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"68'93","reported_seq":1029,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.081037+0000","last_change":"2024-09-16T13:29:07.285303+0000","last_active":"2024-09-16T13:29:09.081037+0000","last_peered":"2024-09-16T13:29:09.081037+0000","last_clean":"2024-09-16T13:29:09.081037+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:29:09.081037+0000","last_undegraded":"2024-09-16T13:29:09.081037+0000","last_fullsized":"2024-09-16T13:29:09.081037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080754000000000004,"stat_sum":{"num_bytes":13548438,"num_objects":29,"num_object_clones":7,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":11,"num_read":1107,"num_read_kb":16627,"num_write":96,"num_write_kb":10358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'171","reported_seq":597,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.240551+0000","last_change":"2024-09-16T13:28:30.062793+0000","last_active":"2024-09-16T13:29:06.240551+0000","last_peered":"2024-09-16T13:29:06.240551+0000","last_clean":"2024-09-16T13:29:06.240551+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:29:06.240551+0000","last_undegraded":"2024-09-16T13:29:06.240551+0000","last_fullsized":"2024-09-16T13:29:06.240551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039817100000000001,"stat_sum":{"num_bytes":10940975,"num_objects":22,"num_object_clones":3,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":4,"num_read":1038,"num_read_kb":3299,"num_write":205,"num_write_kb":3957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'295","reported_seq":1879,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T13:29:09.034566+0000","last_change":"2024-09-16T13:29:09.034797+0000","last_active":"2024-09-16T13:29:09.034566+0000","last_peered":"2024-09-16T13:29:09.034566+0000","last_clean":"2024-09-16T13:29:05.419349+0000","last_became_active":"2024-09-16T13:29:09.034232+0000","last_became_peered":"2024-09-16T13:29:09.034232+0000","last_unstale":"2024-09-16T13:29:09.034566+0000","last_undegraded":"2024-09-16T13:29:08.178202+0000","last_fullsized":"2024-09-16T13:29:08.177268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:33.078269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054,"stat_sum":{"num_bytes":64822358,"num_objects":37,"num_object_clones":8,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":13,"num_read":2414,"num_read_kb":18680,"num_write":505,"num_write_kb":10957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8196910,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":37}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"68'206","reported_seq":1478,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.047545+0000","last_change":"2024-09-16T13:28:28.984368+0000","last_active":"2024-09-16T13:29:09.047545+0000","last_peered":"2024-09-16T13:29:09.047545+0000","last_clean":"2024-09-16T13:29:09.047545+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:29:09.047545+0000","last_undegraded":"2024-09-16T13:29:09.047545+0000","last_fullsized":"2024-09-16T13:29:09.047545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000271229,"stat_sum":{"num_bytes":7407768,"num_objects":35,"num_object_clones":9,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":16,"num_read":2477,"num_read_kb":6649,"num_write":270,"num_write_kb":3986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"68'109","reported_seq":672,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.098431+0000","last_change":"2024-09-16T13:28:28.979874+0000","last_active":"2024-09-16T13:29:09.098431+0000","last_peered":"2024-09-16T13:29:09.098431+0000","last_clean":"2024-09-16T13:29:09.098431+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:29:09.098431+0000","last_undegraded":"2024-09-16T13:29:09.098431+0000","last_fullsized":"2024-09-16T13:29:09.098431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000816496,"stat_sum":{"num_bytes":40087023,"num_objects":40,"num_object_clones":12,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":14,"num_read":570,"num_read_kb":12285,"num_write":123,"num_write_kb":4524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"66'381","reported_seq":912,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.242388+0000","last_change":"2024-09-16T13:28:28.979687+0000","last_active":"2024-09-16T13:29:06.242388+0000","last_peered":"2024-09-16T13:29:06.242388+0000","last_clean":"2024-09-16T13:29:06.242388+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:29:06.242388+0000","last_undegraded":"2024-09-16T13:29:06.242388+0000","last_fullsized":"2024-09-16T13:29:06.242388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035537999999999998,"stat_sum":{"num_bytes":4915311,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2551,"num_read_kb":4656,"num_write":590,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"63'185","reported_seq":2286,"reported_epoch":68,"state":"active+undersized+remapped","last_fresh":"2024-09-16T13:29:09.125200+0000","last_change":"2024-09-16T13:29:09.125428+0000","last_active":"2024-09-16T13:29:09.125200+0000","last_peered":"2024-09-16T13:29:09.125200+0000","last_clean":"2024-09-16T13:29:05.428859+0000","last_became_active":"2024-09-16T13:29:09.032930+0000","last_became_peered":"2024-09-16T13:29:09.032930+0000","last_unstale":"2024-09-16T13:29:09.125200+0000","last_undegraded":"2024-09-16T13:29:09.125200+0000","last_fullsized":"2024-09-16T13:29:08.179909+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:41.653380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035811000000000003,"stat_sum":{"num_bytes":10179226,"num_objects":29,"num_object_clones":7,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":10,"num_read":2251,"num_read_kb":4174,"num_write":255,"num_write_kb":2778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3","objects":29}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'349","reported_seq":1085,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.146916+0000","last_change":"2024-09-16T13:28:28.983418+0000","last_active":"2024-09-16T13:29:09.146916+0000","last_peered":"2024-09-16T13:29:09.146916+0000","last_clean":"2024-09-16T13:29:09.146916+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:29:09.146916+0000","last_undegraded":"2024-09-16T13:29:09.146916+0000","last_fullsized":"2024-09-16T13:29:09.146916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044577700000000001,"stat_sum":{"num_bytes":8485106,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":3015,"num_read_kb":7245,"num_write":549,"num_write_kb":3750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":77,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.182225+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:29:07.182225+0000","last_peered":"2024-09-16T13:29:07.182225+0000","last_clean":"2024-09-16T13:29:07.182225+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:29:07.182225+0000","last_undegraded":"2024-09-16T13:29:07.182225+0000","last_fullsized":"2024-09-16T13:29:07.182225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'201","reported_seq":825,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.093840+0000","last_change":"2024-09-16T13:28:28.979658+0000","last_active":"2024-09-16T13:29:09.093840+0000","last_peered":"2024-09-16T13:29:09.093840+0000","last_clean":"2024-09-16T13:29:09.093840+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:29:09.093840+0000","last_undegraded":"2024-09-16T13:29:09.093840+0000","last_fullsized":"2024-09-16T13:29:09.093840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066082900000000004,"stat_sum":{"num_bytes":17269006,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":10,"num_read":1109,"num_read_kb":16640,"num_write":314,"num_write_kb":11048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.179071+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:06.179071+0000","last_peered":"2024-09-16T13:29:06.179071+0000","last_clean":"2024-09-16T13:29:06.179071+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:29:06.179071+0000","last_undegraded":"2024-09-16T13:29:06.179071+0000","last_fullsized":"2024-09-16T13:29:06.179071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"68'152","reported_seq":1104,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T13:29:09.057875+0000","last_change":"2024-09-16T13:29:09.038157+0000","last_active":"2024-09-16T13:29:09.057875+0000","last_peered":"2024-09-16T13:29:09.057875+0000","last_clean":"2024-09-16T13:29:05.670886+0000","last_became_active":"2024-09-16T13:29:09.033676+0000","last_became_peered":"2024-09-16T13:29:09.033676+0000","last_unstale":"2024-09-16T13:29:09.057875+0000","last_undegraded":"2024-09-16T13:29:08.181226+0000","last_fullsized":"2024-09-16T13:29:08.180058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:29:05.796409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043631399999999998,"stat_sum":{"num_bytes":12530600,"num_objects":35,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":11,"num_read":1232,"num_read_kb":5296,"num_write":214,"num_write_kb":2441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1074310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":32},{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.183410+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:29:06.183410+0000","last_peered":"2024-09-16T13:29:06.183410+0000","last_clean":"2024-09-16T13:29:06.183410+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:29:06.183410+0000","last_undegraded":"2024-09-16T13:29:06.183410+0000","last_fullsized":"2024-09-16T13:29:06.183410+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'305","reported_seq":2111,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.178579+0000","last_change":"2024-09-16T13:28:30.062555+0000","last_active":"2024-09-16T13:29:06.178579+0000","last_peered":"2024-09-16T13:29:06.178579+0000","last_clean":"2024-09-16T13:29:06.178579+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:29:06.178579+0000","last_undegraded":"2024-09-16T13:29:06.178579+0000","last_fullsized":"2024-09-16T13:29:06.178579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000230199,"stat_sum":{"num_bytes":47005908,"num_objects":38,"num_object_clones":10,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":11,"num_read":2750,"num_read_kb":42493,"num_write":455,"num_write_kb":27202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.179846+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:06.179846+0000","last_peered":"2024-09-16T13:29:06.179846+0000","last_clean":"2024-09-16T13:29:06.179846+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:29:06.179846+0000","last_undegraded":"2024-09-16T13:29:06.179846+0000","last_fullsized":"2024-09-16T13:29:06.179846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"66'73","reported_seq":258,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.235693+0000","last_change":"2024-09-16T13:28:28.979887+0000","last_active":"2024-09-16T13:29:06.235693+0000","last_peered":"2024-09-16T13:29:06.235693+0000","last_clean":"2024-09-16T13:29:06.235693+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:29:06.235693+0000","last_undegraded":"2024-09-16T13:29:06.235693+0000","last_fullsized":"2024-09-16T13:29:06.235693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053138000000000003,"stat_sum":{"num_bytes":8209115,"num_objects":27,"num_object_clones":4,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":6,"num_read":200,"num_read_kb":4199,"num_write":91,"num_write_kb":4858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:08.106971+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:29:08.106971+0000","last_peered":"2024-09-16T13:29:08.106971+0000","last_clean":"2024-09-16T13:29:08.106971+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:29:08.106971+0000","last_undegraded":"2024-09-16T13:29:08.106971+0000","last_fullsized":"2024-09-16T13:29:08.106971+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":62,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.175420+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:29:07.175420+0000","last_peered":"2024-09-16T13:29:07.175420+0000","last_clean":"2024-09-16T13:29:07.175420+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:29:07.175420+0000","last_undegraded":"2024-09-16T13:29:07.175420+0000","last_fullsized":"2024-09-16T13:29:07.175420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'109","reported_seq":2323,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.241959+0000","last_change":"2024-09-16T13:28:28.979855+0000","last_active":"2024-09-16T13:29:06.241959+0000","last_peered":"2024-09-16T13:29:06.241959+0000","last_clean":"2024-09-16T13:29:06.241959+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:29:06.241959+0000","last_undegraded":"2024-09-16T13:29:06.241959+0000","last_fullsized":"2024-09-16T13:29:06.241959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039579799999999999,"stat_sum":{"num_bytes":24596494,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":2263,"num_read_kb":15432,"num_write":124,"num_write_kb":3218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":49,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.179229+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:29:06.179229+0000","last_peered":"2024-09-16T13:29:06.179229+0000","last_clean":"2024-09-16T13:29:06.179229+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:29:06.179229+0000","last_undegraded":"2024-09-16T13:29:06.179229+0000","last_fullsized":"2024-09-16T13:29:06.179229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.182687+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:29:06.182687+0000","last_peered":"2024-09-16T13:29:06.182687+0000","last_clean":"2024-09-16T13:29:06.182687+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:29:06.182687+0000","last_undegraded":"2024-09-16T13:29:06.182687+0000","last_fullsized":"2024-09-16T13:29:06.182687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'150","reported_seq":522,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.215693+0000","last_change":"2024-09-16T13:28:28.980209+0000","last_active":"2024-09-16T13:29:06.215693+0000","last_peered":"2024-09-16T13:29:06.215693+0000","last_clean":"2024-09-16T13:29:06.215693+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:29:06.215693+0000","last_undegraded":"2024-09-16T13:29:06.215693+0000","last_fullsized":"2024-09-16T13:29:06.215693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069467499999999996,"stat_sum":{"num_bytes":50890972,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":436,"num_read_kb":5000,"num_write":191,"num_write_kb":4439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"53'1","reported_seq":51,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.178468+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:29:06.178468+0000","last_peered":"2024-09-16T13:29:06.178468+0000","last_clean":"2024-09-16T13:29:06.178468+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:29:06.178468+0000","last_undegraded":"2024-09-16T13:29:06.178468+0000","last_fullsized":"2024-09-16T13:29:06.178468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.184407+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:29:07.184407+0000","last_peered":"2024-09-16T13:29:07.184407+0000","last_clean":"2024-09-16T13:29:07.184407+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:29:07.184407+0000","last_undegraded":"2024-09-16T13:29:07.184407+0000","last_fullsized":"2024-09-16T13:29:07.184407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'157","reported_seq":1020,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.241919+0000","last_change":"2024-09-16T13:28:30.062973+0000","last_active":"2024-09-16T13:29:06.241919+0000","last_peered":"2024-09-16T13:29:06.241919+0000","last_clean":"2024-09-16T13:29:06.241919+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:29:06.241919+0000","last_undegraded":"2024-09-16T13:29:06.241919+0000","last_fullsized":"2024-09-16T13:29:06.241919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049121500000000003,"stat_sum":{"num_bytes":36496131,"num_objects":27,"num_object_clones":5,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":7,"num_read":1871,"num_read_kb":13820,"num_write":205,"num_write_kb":15492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"53'1","reported_seq":52,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.178415+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:29:07.178415+0000","last_peered":"2024-09-16T13:29:07.178415+0000","last_clean":"2024-09-16T13:29:07.178415+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:29:07.178415+0000","last_undegraded":"2024-09-16T13:29:07.178415+0000","last_fullsized":"2024-09-16T13:29:07.178415+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:20:30.803710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":61,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.175512+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:29:07.175512+0000","last_peered":"2024-09-16T13:29:07.175512+0000","last_clean":"2024-09-16T13:29:07.175512+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:29:07.175512+0000","last_undegraded":"2024-09-16T13:29:07.175512+0000","last_fullsized":"2024-09-16T13:29:07.175512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:07.190298+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:29:07.190298+0000","last_peered":"2024-09-16T13:29:07.190298+0000","last_clean":"2024-09-16T13:29:07.190298+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:29:07.190298+0000","last_undegraded":"2024-09-16T13:29:07.190298+0000","last_fullsized":"2024-09-16T13:29:07.190298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"66'167","reported_seq":1192,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.251451+0000","last_change":"2024-09-16T13:28:28.989279+0000","last_active":"2024-09-16T13:29:06.251451+0000","last_peered":"2024-09-16T13:29:06.251451+0000","last_clean":"2024-09-16T13:29:06.251451+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:29:06.251451+0000","last_undegraded":"2024-09-16T13:29:06.251451+0000","last_fullsized":"2024-09-16T13:29:06.251451+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024121900000000001,"stat_sum":{"num_bytes":24620338,"num_objects":30,"num_object_clones":4,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":7,"num_read":1550,"num_read_kb":10829,"num_write":169,"num_write_kb":5524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"68'252","reported_seq":3924,"reported_epoch":68,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T13:29:09.151516+0000","last_change":"2024-09-16T13:29:09.037403+0000","last_active":"2024-09-16T13:29:09.151516+0000","last_peered":"2024-09-16T13:29:09.151516+0000","last_clean":"2024-09-16T13:29:05.422813+0000","last_became_active":"2024-09-16T13:29:09.033711+0000","last_became_peered":"2024-09-16T13:29:09.033711+0000","last_unstale":"2024-09-16T13:29:09.151516+0000","last_undegraded":"2024-09-16T13:29:08.177756+0000","last_fullsized":"2024-09-16T13:29:08.177057+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:38:29.044575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024448900000000002,"stat_sum":{"num_bytes":31100831,"num_objects":34,"num_object_clones":8,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":8,"num_read":4769,"num_read_kb":16325,"num_write":252,"num_write_kb":7111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1503462,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5","objects":1},{"shards":"5,6","objects":33}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"63'114","reported_seq":1739,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:29:08.106850+0000","last_change":"2024-09-16T13:28:28.979223+0000","last_active":"2024-09-16T13:29:08.106850+0000","last_peered":"2024-09-16T13:29:08.106850+0000","last_clean":"2024-09-16T13:29:08.106850+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:29:08.106850+0000","last_undegraded":"2024-09-16T13:29:08.106850+0000","last_fullsized":"2024-09-16T13:29:08.106850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039311499999999998,"stat_sum":{"num_bytes":53771045,"num_objects":32,"num_object_clones":10,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":9,"num_read":1728,"num_read_kb":13269,"num_write":156,"num_write_kb":5804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"66'156","reported_seq":2378,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.245109+0000","last_change":"2024-09-16T13:28:28.990059+0000","last_active":"2024-09-16T13:29:06.245109+0000","last_peered":"2024-09-16T13:29:06.245109+0000","last_clean":"2024-09-16T13:29:06.245109+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:29:06.245109+0000","last_undegraded":"2024-09-16T13:29:06.245109+0000","last_fullsized":"2024-09-16T13:29:06.245109+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086978299999999995,"stat_sum":{"num_bytes":30177803,"num_objects":32,"num_object_clones":8,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":9,"num_read":2617,"num_read_kb":35029,"num_write":202,"num_write_kb":22979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"65'95","reported_seq":3171,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.178482+0000","last_change":"2024-09-16T13:28:30.062577+0000","last_active":"2024-09-16T13:29:06.178482+0000","last_peered":"2024-09-16T13:29:06.178482+0000","last_clean":"2024-09-16T13:29:06.178482+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:29:06.178482+0000","last_undegraded":"2024-09-16T13:29:06.178482+0000","last_fullsized":"2024-09-16T13:29:06.178482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029103499999999998,"stat_sum":{"num_bytes":54028219,"num_objects":27,"num_object_clones":7,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":8,"num_read":3083,"num_read_kb":23418,"num_write":149,"num_write_kb":15830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'229","reported_seq":3601,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.076554+0000","last_change":"2024-09-16T13:28:28.979711+0000","last_active":"2024-09-16T13:29:09.076554+0000","last_peered":"2024-09-16T13:29:09.076554+0000","last_clean":"2024-09-16T13:29:09.076554+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:29:09.076554+0000","last_undegraded":"2024-09-16T13:29:09.076554+0000","last_fullsized":"2024-09-16T13:29:09.076554+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071623100000000001,"stat_sum":{"num_bytes":38240380,"num_objects":32,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":13,"num_read":4929,"num_read_kb":11308,"num_write":315,"num_write_kb":2498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"68'163","reported_seq":1296,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.073978+0000","last_change":"2024-09-16T13:28:28.979910+0000","last_active":"2024-09-16T13:29:09.073978+0000","last_peered":"2024-09-16T13:29:09.073978+0000","last_clean":"2024-09-16T13:29:09.073978+0000","last_became_active":"2024-09-16T13:28:26.987044+0000","last_became_peered":"2024-09-16T13:28:26.987044+0000","last_unstale":"2024-09-16T13:29:09.073978+0000","last_undegraded":"2024-09-16T13:29:09.073978+0000","last_fullsized":"2024-09-16T13:29:09.073978+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:11.809480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084175000000000005,"stat_sum":{"num_bytes":27785225,"num_objects":37,"num_object_clones":9,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":14,"num_read":1285,"num_read_kb":7200,"num_write":171,"num_write_kb":4077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'117","reported_seq":475,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:29:09.078422+0000","last_change":"2024-09-16T13:28:28.980116+0000","last_active":"2024-09-16T13:29:09.078422+0000","last_peered":"2024-09-16T13:29:09.078422+0000","last_clean":"2024-09-16T13:29:09.078422+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:29:09.078422+0000","last_undegraded":"2024-09-16T13:29:09.078422+0000","last_fullsized":"2024-09-16T13:29:09.078422+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010142790000000001,"stat_sum":{"num_bytes":15304503,"num_objects":30,"num_object_clones":6,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":17,"num_read":377,"num_read_kb":8805,"num_write":115,"num_write_kb":5791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"65'450","reported_seq":1761,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:29:06.182871+0000","last_change":"2024-09-16T13:28:58.119992+0000","last_active":"2024-09-16T13:29:06.182871+0000","last_peered":"2024-09-16T13:29:06.182871+0000","last_clean":"2024-09-16T13:29:06.182871+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:29:06.182871+0000","last_undegraded":"2024-09-16T13:29:06.182871+0000","last_fullsized":"2024-09-16T13:29:06.182871+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074187000000000005,"stat_sum":{"num_bytes":25965948,"num_objects":36,"num_object_clones":12,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":15,"num_read":2976,"num_read_kb":6300,"num_write":709,"num_write_kb":5286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":905066786,"num_objects":968,"num_object_clones":222,"num_object_copies":1936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":302,"num_read":67509,"num_read_kb":376943,"num_write":7810,"num_write_kb":217037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":33666691,"num_keys_recovered":48,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":41439232,"data_stored":429178758,"data_compressed":8945651,"data_compressed_allocated":33263616,"data_compressed_original":424030174,"omap_allocated":0,"internal_metadata":0},"log_size":5826,"ondisk_log_size":5826,"up":64,"acting":59,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303020,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068820,"statfs":{"total":96636764160,"available":96326471680,"internally_reserved":0,"allocated":1531904,"data_stored":10161552,"data_compressed":100514,"data_compressed_allocated":712704,"data_compressed_original":9777152,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.869,"5min":1.869,"15min":1.869},"last":0.503},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.315}]},{"osd":5,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.371}]},{"osd":6,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41299999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312320,"kb_used_data":6644,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059520,"statfs":{"total":96636764160,"available":96316948480,"internally_reserved":0,"allocated":6803456,"data_stored":69537603,"data_compressed":1173929,"data_compressed_allocated":4931584,"data_compressed_original":68444808,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.169,"5min":2.169,"15min":2.169},"last":0.39100000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.379}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.32500000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.311},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34599999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.433}]}]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312112,"kb_used_data":6436,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059728,"statfs":{"total":96636764160,"available":96317161472,"internally_reserved":0,"allocated":6590464,"data_stored":49482376,"data_compressed":1616432,"data_compressed_allocated":4558848,"data_compressed_original":48295271,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.495},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38700000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.317},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34799999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.436},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42399999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316740,"kb_used_data":6900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94055100,"statfs":{"total":96636764160,"available":96312422400,"internally_reserved":0,"allocated":7065600,"data_stored":51855517,"data_compressed":1408751,"data_compressed_allocated":4665344,"data_compressed_original":50514831,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,13],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.33600000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.437},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.42499999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.495}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.51100000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311864,"kb_used_data":6188,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94059976,"statfs":{"total":96636764160,"available":96317415424,"internally_reserved":0,"allocated":6336512,"data_stored":62644549,"data_compressed":1230677,"data_compressed_allocated":4648960,"data_compressed_original":61748771,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":1.3100000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":1.236}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":1.3500000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":1.2989999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":1.288}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.44}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48199999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304304,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067536,"statfs":{"total":96636764160,"available":96325156864,"internally_reserved":0,"allocated":2854912,"data_stored":16953548,"data_compressed":562700,"data_compressed_allocated":1474560,"data_compressed_original":16228352,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.3390000000000004,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.25700000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":9.4480000000000004,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.308}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.27600000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.4090000000000007,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.5640000000000001,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318948,"kb_used_data":9108,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052892,"statfs":{"total":96636764160,"available":96310161408,"internally_reserved":0,"allocated":9326592,"data_stored":92867889,"data_compressed":1800742,"data_compressed_allocated":6893568,"data_compressed_original":91443200,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.31},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.434}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.45300000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.499}]}]},{"osd":0,"up_from":14,"seq":60129542161,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308088,"kb_used_data":6568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94063752,"statfs":{"total":96636764160,"available":96321282048,"internally_reserved":0,"allocated":6725632,"data_stored":76219311,"data_compressed":977649,"data_compressed_allocated":5062656,"data_compressed_original":75243069,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.373}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5894144,"data_stored":75264643,"data_compressed":972050,"data_compressed_allocated":5025792,"data_compressed_original":74669629,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8548352,"data_stored":92503570,"data_compressed":1800742,"data_compressed_allocated":6893568,"data_compressed_original":91443200,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2080768,"data_stored":16589248,"data_compressed":562700,"data_compressed_allocated":1474560,"data_compressed_original":16228352,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6299648,"data_stored":51589311,"data_compressed":1409007,"data_compressed_allocated":4673536,"data_compressed_original":50613135,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5873664,"data_stored":49642364,"data_compressed":1646651,"data_compressed_allocated":4616192,"data_compressed_original":48819559,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6012928,"data_stored":69166895,"data_compressed":1173929,"data_compressed_allocated":4931584,"data_compressed_original":68444808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":62280249,"data_compressed":1230677,"data_compressed_allocated":4648960,"data_compressed_original":61748771,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167360,"data_stored":12142478,"data_compressed":149895,"data_compressed_allocated":999424,"data_compressed_original":12062720,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:29:11.977 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [2, 0, 1, 5] 2024-09-16T13:29:11.977 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.2', '2', '0', '1', '5'] 2024-09-16T13:29:11.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.2 2 0 1 5 2024-09-16T13:29:12.057 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.058 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.061 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:12.059+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:12.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:12.162 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:12.160+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:12.171 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.171 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:12.262 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:12.261+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:12.283 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone20 from 0xe3a4764 (overlap 0x3b14c4c) to 0x17a1e84 2024-09-16T13:29:12.293 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.293 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:12.363 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:12.361+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:12.425 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-09-16T13:29:12.450 INFO:teuthology.orchestra.run.smithi076.stdout:2448 writesame 0x12e666 thru 0x13a660 data_size 0x3b(0xbffb bytes) 2024-09-16T13:29:12.461 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.461 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.463 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:12.462+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:12.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:12.489 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.489 INFO:teuthology.orchestra.run.smithi076.stdout:2451 writesame 0xdc3ce5c thru 0xdc440a9 data_size 0x3(0x724e bytes) 2024-09-16T13:29:12.510 INFO:teuthology.orchestra.run.smithi076.stdout:2454 write 0x595c59f thru 0x5966e59 (0xa8bb bytes) 2024-09-16T13:29:12.516 INFO:teuthology.orchestra.run.smithi076.stdout:2455 writesame 0x20bf5c2 thru 0x20c3200 data_size 0x3(0x3c3f bytes) 2024-09-16T13:29:12.528 INFO:teuthology.orchestra.run.smithi076.stdout:2456 writesame 0x40a7f03 thru 0x40b3a16 data_size 0xd(0xbb14 bytes) 2024-09-16T13:29:12.539 INFO:teuthology.orchestra.run.smithi076.stdout:2457 read 0x9396fa0 thru 0x939e9aa (0x7a0b bytes) 2024-09-16T13:29:12.539 INFO:teuthology.orchestra.run.smithi076.stdout:2458 write 0x82cc4f8 thru 0x82cf55e (0x3067 bytes) 2024-09-16T13:29:12.552 INFO:teuthology.orchestra.run.smithi076.stdout:2459 trunc from 0xe3a4764 to 0x71ab0e7 2024-09-16T13:29:12.564 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:12.562+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:12.576 INFO:teuthology.orchestra.run.smithi076.stdout:2460 read 0x707788a thru 0x70854a9 (0xdc20 bytes) 2024-09-16T13:29:12.577 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.582 INFO:teuthology.orchestra.run.smithi076.stdout:2462 write 0x7c23553 thru 0x7c29a22 (0x64d0 bytes) 2024-09-16T13:29:12.584 INFO:teuthology.orchestra.run.smithi076.stdout:2463 writesame 0x713dbd9 thru 0x714c898 data_size 0x20(0xecc0 bytes) 2024-09-16T13:29:12.587 INFO:teuthology.orchestra.run.smithi076.stdout:2464 punch from 0x172aa3a to 0x1738c25, (0xe1eb bytes) 2024-09-16T13:29:12.587 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.587 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.589 INFO:teuthology.orchestra.run.smithi076.stdout:2466 trunc from 0x7c29a23 to 0xa086ff8 2024-09-16T13:29:12.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:12.607 INFO:teuthology.orchestra.run.smithi076.stdout:2467 trunc from 0xa086ff8 to 0x39dd9c 2024-09-16T13:29:12.648 INFO:teuthology.orchestra.run.smithi076.stdout:2468 read 0x11fd09 thru 0x121d71 (0x2069 bytes) 2024-09-16T13:29:12.654 INFO:teuthology.orchestra.run.smithi076.stdout:2469 writesame 0xf67e07 thru 0xf69a3e data_size 0x38(0x1c38 bytes) 2024-09-16T13:29:12.665 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:12.663+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:12.667 INFO:teuthology.orchestra.run.smithi076.stdout:2470 write 0x7595058 thru 0x75a058f (0xb538 bytes) 2024-09-16T13:29:12.670 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.670 INFO:teuthology.orchestra.run.smithi076.stdout:2473 read 0x4748c5d thru 0x47580a8 (0xf44c bytes) 2024-09-16T13:29:12.670 INFO:teuthology.orchestra.run.smithi076.stdout:2475 punch from 0x372d84e to 0x373d6c8, (0xfe7a bytes) 2024-09-16T13:29:12.676 INFO:teuthology.orchestra.run.smithi076.stdout:2477 write 0x81b7ac0 thru 0x81be7cd (0x6d0e bytes) 2024-09-16T13:29:12.679 INFO:teuthology.orchestra.run.smithi076.stdout:2478 punch from 0x566708b to 0x566d90d, (0x6882 bytes) 2024-09-16T13:29:12.688 INFO:teuthology.orchestra.run.smithi076.stdout:2480 write 0xaf9abcf thru 0xaf9b994 (0xdc6 bytes) 2024-09-16T13:29:12.690 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.697 INFO:teuthology.orchestra.run.smithi076.stdout:2482 writesame 0xc047129 thru 0xc050735 data_size 0x1(0x960d bytes) 2024-09-16T13:29:12.697 INFO:teuthology.orchestra.run.smithi076.stdout:2483 punch from 0xabf1f83 to 0xabfeb82, (0xcbff bytes) 2024-09-16T13:29:12.697 INFO:teuthology.orchestra.run.smithi076.stdout:2484 writesame 0x85a3acd thru 0x85aa04d data_size 0x1(0x6581 bytes) 2024-09-16T13:29:12.725 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.725 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.734 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:12.747 INFO:teuthology.orchestra.run.smithi076.stdout:2485 write 0x209b900 thru 0x209dce6 (0x23e7 bytes) 2024-09-16T13:29:12.750 INFO:teuthology.orchestra.run.smithi076.stdout:2487 punch from 0x3d8f121 to 0x3d9ba15, (0xc8f4 bytes) 2024-09-16T13:29:12.750 INFO:teuthology.orchestra.run.smithi076.stdout:2488 writesame 0x21a3596 thru 0x21ad5d9 data_size 0x34(0xa044 bytes) 2024-09-16T13:29:12.753 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.754 INFO:teuthology.orchestra.run.smithi076.stdout:2490 read 0x3da43d2 thru 0x3daed27 (0xa956 bytes) 2024-09-16T13:29:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:2491 write 0xcaeab8a thru 0xcaedd9f (0x3216 bytes) 2024-09-16T13:29:12.762 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.762 INFO:teuthology.orchestra.run.smithi076.stdout:2493 read 0x5c6320b thru 0x5c728e4 (0xf6da bytes) 2024-09-16T13:29:12.762 INFO:teuthology.orchestra.run.smithi076.stdout:2494 read 0x985d805 thru 0x9860f73 (0x376f bytes) 2024-09-16T13:29:12.763 INFO:teuthology.orchestra.run.smithi076.stdout:2495 write 0x76a9389 thru 0x76a9cbc (0x934 bytes) 2024-09-16T13:29:12.765 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:12.765 INFO:teuthology.orchestra.run.smithi076.stdout:2497 write 0x3ea61d6 thru 0x3eaf9ff (0x982a bytes) 2024-09-16T13:29:12.765 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:12.763+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:12.779 INFO:teuthology.orchestra.run.smithi076.stdout:2498 trunc from 0xcaedda0 to 0x40baeb1 2024-09-16T13:29:12.788 INFO:teuthology.orchestra.run.smithi076.stdout:2499 read 0x29fd167 thru 0x2a0872e (0xb5c8 bytes) 2024-09-16T13:29:12.788 INFO:teuthology.orchestra.run.smithi076.stdout:2500 clone 22 order 24 su 131072 sc 7 2024-09-16T13:29:12.849 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:12.850 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:12.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:12.866 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:12.864+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:12.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:29:12.967 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:12.965+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:13.000 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:13.000 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:13.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:13.068 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:13.065+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:13.109 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:13.110 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:13.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:13.168 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:13.167+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.2 pg_upmap_items mapping to [2->0,1->5] 2024-09-16T13:29:13.220 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:13.220 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:13.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:13.269 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:13.267+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:13.287 INFO:teuthology.orchestra.run.smithi076.stdout:leaving image image_client.0-clone21 intact 2024-09-16T13:29:13.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:13.370 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:13.368+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:13.380 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:13.380 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:13.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:13.397 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-09-16T13:29:13.465 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.465 INFO:teuthology.orchestra.run.smithi076.stdout:2503 punch from 0x3d8829a to 0x3d8a888, (0x25ee bytes) 2024-09-16T13:29:13.471 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:13.469+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:13.481 INFO:teuthology.orchestra.run.smithi076.stdout:2504 writesame 0x9370567 thru 0x937aedc data_size 0x2(0xa976 bytes) 2024-09-16T13:29:13.489 INFO:teuthology.orchestra.run.smithi076.stdout:2505 write 0xb53f984 thru 0xb543b7d (0x41fa bytes) 2024-09-16T13:29:13.498 INFO:teuthology.orchestra.run.smithi076.stdout:2507 write 0xc01bbbf thru 0xc020502 (0x4944 bytes) 2024-09-16T13:29:13.504 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:13.512 INFO:teuthology.orchestra.run.smithi076.stdout:2508 write 0xeaf79d1 thru 0xeb06fcc (0xf5fc bytes) 2024-09-16T13:29:13.516 INFO:teuthology.orchestra.run.smithi076.stdout:2509 trunc from 0xeb06fcd to 0xcb95ba7 2024-09-16T13:29:13.527 INFO:teuthology.orchestra.run.smithi076.stdout:2510 write 0xc584fc2 thru 0xc58ea91 (0x9ad0 bytes) 2024-09-16T13:29:13.531 INFO:teuthology.orchestra.run.smithi076.stdout:2511 punch from 0xbfc766b to 0xbfd0d5f, (0x96f4 bytes) 2024-09-16T13:29:13.531 INFO:teuthology.orchestra.run.smithi076.stdout:2513 read 0x465c9bf thru 0x4660183 (0x37c5 bytes) 2024-09-16T13:29:13.531 INFO:teuthology.orchestra.run.smithi076.stdout:2514 write 0xb196a3c thru 0xb19d9ac (0x6f71 bytes) 2024-09-16T13:29:13.535 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:13.535 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:13.537 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:13.537 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:13.549 INFO:teuthology.orchestra.run.smithi076.stdout:2515 trunc from 0xcb95ba7 to 0x11ddaf7 2024-09-16T13:29:13.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:13.567 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:13.568 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:13.591 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:13.591 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:13.589+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:13.601 INFO:teuthology.orchestra.run.smithi076.stdout:2518 writesame 0x7d470e5 thru 0x7d4c9cd data_size 0x1b(0x58e9 bytes) 2024-09-16T13:29:13.601 INFO:teuthology.orchestra.run.smithi076.stdout:2520 read 0x4695468 thru 0x469833a (0x2ed3 bytes) 2024-09-16T13:29:13.609 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:13.612 INFO:teuthology.orchestra.run.smithi076.stdout:2521 writesame 0xc4982b6 thru 0xc4a5d21 data_size 0xe(0xda6c bytes) 2024-09-16T13:29:13.612 INFO:teuthology.orchestra.run.smithi076.stdout:2522 read 0x49f6ce3 thru 0x49fee97 (0x81b5 bytes) 2024-09-16T13:29:13.612 INFO:teuthology.orchestra.run.smithi076.stdout:2523 writesame 0x3941c6f thru 0x3948f07 data_size 0x4d(0x7299 bytes) 2024-09-16T13:29:13.621 INFO:teuthology.orchestra.run.smithi076.stdout:2524 write 0xcfa4d36 thru 0xcfb3f53 (0xf21e bytes) 2024-09-16T13:29:13.627 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:13.638 INFO:teuthology.orchestra.run.smithi076.stdout:2525 trunc from 0xcfb3f54 to 0x1f2e6b0 2024-09-16T13:29:13.644 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:13.644 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:13.651 INFO:teuthology.orchestra.run.smithi076.stdout:2526 read 0x3784de thru 0x37e6fd (0x6220 bytes) 2024-09-16T13:29:13.660 INFO:teuthology.orchestra.run.smithi076.stdout:2527 write 0x35883ac thru 0x358e91a (0x656f bytes) 2024-09-16T13:29:13.662 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:13.662 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:13.672 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:13.671+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:13.679 INFO:teuthology.orchestra.run.smithi076.stdout:2528 write 0xe3a20d5 thru 0xe3ab274 (0x91a0 bytes) 2024-09-16T13:29:13.683 INFO:teuthology.orchestra.run.smithi076.stdout:2529 writesame 0xde803ff thru 0xde87b71 data_size 0x1(0x7773 bytes) 2024-09-16T13:29:13.687 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:13.687 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:13.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:13.687 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:3297 4.17 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2069521~10445] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.017249266999997, 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030440354999999999, 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3297 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.688 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667257+0000", 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.667258+0000", 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668628+0000", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001369501 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668651+0000", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3017999999999999e-05 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668658+0000", 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3409999999999997e-06 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668704+0000", 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5479000000000003e-05 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.668825+0000", 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121791 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.669018+0000", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019308300000000001 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.685320+0000", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016301690000000001 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697626+0000", 2024-09-16T13:29:13.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012306375 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697668+0000", 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1189999999999997e-05 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.697698+0000", 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0006999999999999e-05 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.694 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8556 4.0 4:017e3810:::rbd_data.11a91a08d9ca.00000000000000d0:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.743951662999997, 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03408597, 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8556 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940555+0000", 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940556+0000", 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.127e-06 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940560+0000", 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940570+0000", 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0302000000000001e-05 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940574+0000", 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940608+0000", 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3748000000000003e-05 2024-09-16T13:29:13.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.940660+0000", 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1402e-05 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:13.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.942061+0000", 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014009750000000001 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974517+0000", 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032456609999999997 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974529+0000", 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2235000000000001e-05 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974593+0000", 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974641+0000", 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7459000000000001e-05 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8556 4.0 e42/32 ondisk, result = 0)", 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.737827525, 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027841225000000001, 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946679+0000", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946680+0000", 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946683+0000", 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6009999999999999e-06 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946694+0000", 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0949e-05 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.946699+0000", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7740000000000004e-06 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974498+0000", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027798386000000001 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974509+0000", 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1157e-05 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.974520+0000", 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1517e-05 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21651 4.10 4:09794630:::rbd_data.11a98415fd69.0000000000000003:head [delete] snapc c=[c] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.553928388999999, 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024369312000000001, 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21651 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130578+0000", 2024-09-16T13:29:13.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130579+0000", 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9999999999999998e-07 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130581+0000", 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.537e-06 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130587+0000", 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.130591+0000", 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138200+0000", 2024-09-16T13:29:13.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007608325 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138265+0000", 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5007999999999998e-05 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.138437+0000", 2024-09-16T13:29:13.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017266100000000001 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.141654+0000", 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032171859999999999 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154881+0000", 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013226236000000001 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154915+0000", 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4471999999999998e-05 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:43.154947+0000", 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2372000000000001e-05 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49432 4.10 4:09c05623:::rbd_data.11a9b5ab4b7f.0000000000000041:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:13.713 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1326847999999998, 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025173402000000001, 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49432 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:13.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551822+0000", 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551823+0000", 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.376e-06 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551830+0000", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8140000000000003e-06 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551843+0000", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2665e-05 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551848+0000", 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2059999999999998e-06 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551886+0000", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8642e-05 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.551976+0000", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9247999999999997e-05 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.553110+0000", 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001134471 2024-09-16T13:29:13.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576835+0000", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023724463000000001 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576884+0000", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9734999999999997e-05 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576951+0000", 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6120999999999999e-05 2024-09-16T13:29:13.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576995+0000", 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4660999999999998e-05 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:49432 4.10 e64/32 ondisk, result = 0)", 2024-09-16T13:29:13.719 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1282408890000006, 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020739172, 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556266+0000", 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3200000000000005e-07 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556269+0000", 2024-09-16T13:29:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0510000000000001e-06 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556279+0000", 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4760000000000004e-06 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.556283+0000", 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576867+0000", 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020583946999999998 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.576876+0000", 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6360000000000006e-06 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.577005+0000", 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012836999999999999 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4422580050000002, 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:13.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:13.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:29:13.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:29:13.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4421955320000004, 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:13.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:29:13.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4421508789999997, 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:29:13.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:13.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:29:13.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:13.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4420777779999998, 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:13.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:13.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:13.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:29:13.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:29:13.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:29:13.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4420592719999998, 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:13.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:13.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:29:13.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:29:13.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:13.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4418853199999999, 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:13.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4418467660000003, 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:13.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:13.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:29:13.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4408880719999999, 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:29:13.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:13.776 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4408302749999997, 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:13.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:29:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4407363990000004, 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:29:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:29:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4407084100000001, 2024-09-16T13:29:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:29:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:29:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4394343840000001, 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:29:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.4394111379999996, 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:29:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.7120649889999999, 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:29:13.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:13.815 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:13.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:13.818 INFO:teuthology.orchestra.run.smithi076.stdout:2530 punch from 0x8c36e31 to 0x8c44263, (0xd432 bytes) 2024-09-16T13:29:13.818 INFO:teuthology.orchestra.run.smithi076.stdout:2532 writesame 0x9c29cd9 thru 0x9c3683b data_size 0x1(0xcb63 bytes) 2024-09-16T13:29:13.818 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:13.815+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:13.831 INFO:teuthology.orchestra.run.smithi076.stdout:2533 read 0x93dea47 thru 0x93ea459 (0xba13 bytes) 2024-09-16T13:29:13.840 INFO:teuthology.orchestra.run.smithi076.stdout:2534 trunc from 0xe3ab275 to 0x5df1e1e 2024-09-16T13:29:13.866 INFO:teuthology.orchestra.run.smithi076.stdout:2536 write 0xd9fa7b9 thru 0xda0265f (0x7ea7 bytes) 2024-09-16T13:29:13.869 INFO:teuthology.orchestra.run.smithi076.stdout:2537 read 0xc32bea1 thru 0xc339db0 (0xdf10 bytes) 2024-09-16T13:29:13.870 INFO:teuthology.orchestra.run.smithi076.stdout:2539 read 0x21d52fa thru 0x21e0caf (0xb9b6 bytes) 2024-09-16T13:29:13.870 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.870 INFO:teuthology.orchestra.run.smithi076.stdout:2542 read 0xb3aa582 thru 0xb3ac281 (0x1d00 bytes) 2024-09-16T13:29:13.870 INFO:teuthology.orchestra.run.smithi076.stdout:2543 read 0xb3cdb72 thru 0xb3d7229 (0x96b8 bytes) 2024-09-16T13:29:13.871 INFO:teuthology.orchestra.run.smithi076.stdout:2544 punch from 0x53894b4 to 0x538ea34, (0x5580 bytes) 2024-09-16T13:29:13.871 INFO:teuthology.orchestra.run.smithi076.stdout:2545 read 0xa7603e4 thru 0xa765950 (0x556d bytes) 2024-09-16T13:29:13.871 INFO:teuthology.orchestra.run.smithi076.stdout:2549 write 0xb127b23 thru 0xb12a75d (0x2c3b bytes) 2024-09-16T13:29:13.875 INFO:teuthology.orchestra.run.smithi076.stdout:2552 trunc from 0xda02660 to 0xb9ea9a8 2024-09-16T13:29:13.882 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.882 INFO:teuthology.orchestra.run.smithi076.stdout:2556 write 0x7356530 thru 0x735a04c (0x3b1d bytes) 2024-09-16T13:29:13.886 INFO:teuthology.orchestra.run.smithi076.stdout:2557 read 0x4c185d9 thru 0x4c1b450 (0x2e78 bytes) 2024-09-16T13:29:13.887 INFO:teuthology.orchestra.run.smithi076.stdout:2558 write 0x91505cc thru 0x9158c38 (0x866d bytes) 2024-09-16T13:29:13.889 INFO:teuthology.orchestra.run.smithi076.stdout:2560 write 0x27311ad thru 0x273a16d (0x8fc1 bytes) 2024-09-16T13:29:13.891 INFO:teuthology.orchestra.run.smithi076.stdout:2561 writesame 0x96e3775 thru 0x96e7710 data_size 0x3b(0x3f9c bytes) 2024-09-16T13:29:13.893 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.893 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.893 INFO:teuthology.orchestra.run.smithi076.stdout:2565 read 0xb04bd1 thru 0xb13dc2 (0xf1f2 bytes) 2024-09-16T13:29:13.893 INFO:teuthology.orchestra.run.smithi076.stdout:2566 write 0x80fb796 thru 0x810aa0f (0xf27a bytes) 2024-09-16T13:29:13.897 INFO:teuthology.orchestra.run.smithi076.stdout:2567 read 0x2c0e4b thru 0x2cda3b (0xcbf1 bytes) 2024-09-16T13:29:13.897 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.898 INFO:teuthology.orchestra.run.smithi076.stdout:2572 writesame 0x51d493e thru 0x51d6536 data_size 0x1(0x1bf9 bytes) 2024-09-16T13:29:13.901 INFO:teuthology.orchestra.run.smithi076.stdout:2574 writesame 0x597c904 thru 0x5981763 data_size 0x30(0x4e60 bytes) 2024-09-16T13:29:13.903 INFO:teuthology.orchestra.run.smithi076.stdout:2575 read 0x625b8b3 thru 0x625c0e9 (0x837 bytes) 2024-09-16T13:29:13.903 INFO:teuthology.orchestra.run.smithi076.stdout:2576 punch from 0x69dc139 to 0x69e1c8d, (0x5b54 bytes) 2024-09-16T13:29:13.903 INFO:teuthology.orchestra.run.smithi076.stdout:2578 read 0x358f73 thru 0x364f50 (0xbfde bytes) 2024-09-16T13:29:13.904 INFO:teuthology.orchestra.run.smithi076.stdout:2579 punch from 0x9ad022f to 0x9ad65db, (0x63ac bytes) 2024-09-16T13:29:13.904 INFO:teuthology.orchestra.run.smithi076.stdout:2580 write 0x14adb8e thru 0x14b0eb1 (0x3324 bytes) 2024-09-16T13:29:13.905 INFO:teuthology.orchestra.run.smithi076.stdout:2581 writesame 0x3d30bd2 thru 0x3d3ba67 data_size 0x75(0xae96 bytes) 2024-09-16T13:29:13.907 INFO:teuthology.orchestra.run.smithi076.stdout:2582 writesame 0x4a154b3 thru 0x4a1b415 data_size 0x1(0x5f63 bytes) 2024-09-16T13:29:13.918 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:13.916+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:13.922 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:13.922 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:13.922 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:13.922 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:13.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:13.948 INFO:teuthology.orchestra.run.smithi076.stdout:2583 read 0x544e126 thru 0x545863f (0xa51a bytes) 2024-09-16T13:29:13.948 INFO:teuthology.orchestra.run.smithi076.stdout:2584 read 0x9051cb3 thru 0x90606e1 (0xea2f bytes) 2024-09-16T13:29:13.949 INFO:teuthology.orchestra.run.smithi076.stdout:2585 write 0x5b65c7c thru 0x5b6c6ba (0x6a3f bytes) 2024-09-16T13:29:13.963 INFO:teuthology.orchestra.run.smithi076.stdout:2586 trunc from 0xb9ea9a8 to 0x11d223a 2024-09-16T13:29:13.974 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.974 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:13.974 INFO:teuthology.orchestra.run.smithi076.stdout:2590 read 0x9e4e24 thru 0x9e9b52 (0x4d2f bytes) 2024-09-16T13:29:13.975 INFO:teuthology.orchestra.run.smithi076.stdout:2591 punch from 0x10d64ad to 0x10d79be, (0x1511 bytes) 2024-09-16T13:29:13.975 INFO:teuthology.orchestra.run.smithi076.stdout:2592 punch from 0xe8c7ba to 0xe95ab8, (0x92fe bytes) 2024-09-16T13:29:13.975 INFO:teuthology.orchestra.run.smithi076.stdout:2593 punch from 0x939cd5 to 0x948f82, (0xf2ad bytes) 2024-09-16T13:29:13.976 INFO:teuthology.orchestra.run.smithi076.stdout:2595 writesame 0xe913e7 thru 0xe9b642 data_size 0x2(0xa25c bytes) 2024-09-16T13:29:14.003 INFO:teuthology.orchestra.run.smithi076.stdout:2596 write 0x7a237b8 thru 0x7a2a7bb (0x7004 bytes) 2024-09-16T13:29:14.005 INFO:teuthology.orchestra.run.smithi076.stdout:2600 write 0x174f0bf thru 0x1755a01 (0x6943 bytes) 2024-09-16T13:29:14.007 INFO:teuthology.orchestra.run.smithi076.stdout:2601 read 0x4814632 thru 0x4817e27 (0x37f6 bytes) 2024-09-16T13:29:14.008 INFO:teuthology.orchestra.run.smithi076.stdout:2602 write 0x76cad93 thru 0x76cf4a7 (0x4715 bytes) 2024-09-16T13:29:14.011 INFO:teuthology.orchestra.run.smithi076.stdout:2603 trunc from 0x7a2a7bc to 0x6c93967 2024-09-16T13:29:14.019 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:14.017+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:2604 read 0x6884092 thru 0x689022c (0xc19b bytes) 2024-09-16T13:29:14.022 INFO:teuthology.orchestra.run.smithi076.stdout:2605 read 0x63efe4d thru 0x63f3d8c (0x3f40 bytes) 2024-09-16T13:29:14.023 INFO:teuthology.orchestra.run.smithi076.stdout:2606 write 0x28632bf thru 0x286861a (0x535c bytes) 2024-09-16T13:29:14.025 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.025 INFO:teuthology.orchestra.run.smithi076.stdout:2608 punch from 0x1fd7150 to 0x1fe68a1, (0xf751 bytes) 2024-09-16T13:29:14.031 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.031 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:14.031 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:14.031 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:14.032 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.032 INFO:teuthology.orchestra.run.smithi076.stdout:2609 write 0x89a929a thru 0x89ab7a3 (0x250a bytes) 2024-09-16T13:29:14.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:14.043 INFO:teuthology.orchestra.run.smithi076.stdout:2610 writesame 0xbf8b92a thru 0xbf8e42c data_size 0x4d(0x2b03 bytes) 2024-09-16T13:29:14.043 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.043 INFO:teuthology.orchestra.run.smithi076.stdout:2613 read 0x2d5566d thru 0x2d59145 (0x3ad9 bytes) 2024-09-16T13:29:14.044 INFO:teuthology.orchestra.run.smithi076.stdout:2615 punch from 0x523f522 to 0x524b92c, (0xc40a bytes) 2024-09-16T13:29:14.044 INFO:teuthology.orchestra.run.smithi076.stdout:2617 write 0x489d310 thru 0x48a700f (0x9d00 bytes) 2024-09-16T13:29:14.046 INFO:teuthology.orchestra.run.smithi076.stdout:2619 write 0x672540c thru 0x672e72f (0x9324 bytes) 2024-09-16T13:29:14.048 INFO:teuthology.orchestra.run.smithi076.stdout:2621 writesame 0x402158e thru 0x4024237 data_size 0x1(0x2caa bytes) 2024-09-16T13:29:14.066 INFO:teuthology.orchestra.run.smithi076.stdout:2623 trunc from 0xbf8e42d to 0xbbdee8b 2024-09-16T13:29:14.072 INFO:teuthology.orchestra.run.smithi076.stdout:2624 read 0x6c7f6b9 thru 0x6c7f701 (0x49 bytes) 2024-09-16T13:29:14.072 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.072 INFO:teuthology.orchestra.run.smithi076.stdout:2626 trunc from 0xbbdee8b to 0xb6f70d3 2024-09-16T13:29:14.085 INFO:teuthology.orchestra.run.smithi076.stdout:2627 write 0xd6526d6 thru 0xd6579ef (0x531a bytes) 2024-09-16T13:29:14.087 INFO:teuthology.orchestra.run.smithi076.stdout:2628 read 0xc3fe5e6 thru 0xc409499 (0xaeb4 bytes) 2024-09-16T13:29:14.087 INFO:teuthology.orchestra.run.smithi076.stdout:2630 read 0x4323f40 thru 0x4331eed (0xdfae bytes) 2024-09-16T13:29:14.088 INFO:teuthology.orchestra.run.smithi076.stdout:2631 read 0x51d5b28 thru 0x51d61a1 (0x67a bytes) 2024-09-16T13:29:14.088 INFO:teuthology.orchestra.run.smithi076.stdout:2632 write 0xc6b3d64 thru 0xc6c30c2 (0xf35f bytes) 2024-09-16T13:29:14.103 INFO:teuthology.orchestra.run.smithi076.stdout:2633 trunc from 0xd6579f0 to 0x5c83b81 2024-09-16T13:29:14.116 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.116 INFO:teuthology.orchestra.run.smithi076.stdout:2639 read 0x33733d5 thru 0x33802c1 (0xceed bytes) 2024-09-16T13:29:14.119 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:14.118+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:14.126 INFO:teuthology.orchestra.run.smithi076.stdout:2640 write 0x81c1d49 thru 0x81c20d8 (0x390 bytes) 2024-09-16T13:29:14.130 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.130 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.130 INFO:teuthology.orchestra.run.smithi076.stdout:2643 write 0x59ed63d thru 0x59f5cd5 (0x8699 bytes) 2024-09-16T13:29:14.143 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.143 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:14.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:14.143 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3132 4.9 4:92394234:::rbd_data.11a976a8123d.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 4368762~22099] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.532534005000002, 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024228047999999999, 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3132 2024-09-16T13:29:14.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608310+0000", 2024-09-16T13:29:14.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.608311+0000", 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4300000000000002e-07 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614081+0000", 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057699559999999997 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614094+0000", 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2672e-05 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614100+0000", 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.071e-06 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614155+0000", 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5229999999999998e-05 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614248+0000", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2958e-05 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.614457+0000", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020964000000000001 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.629775+0000", 2024-09-16T13:29:14.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015317084 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632471+0000", 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026965330000000001 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632510+0000", 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8831000000000001e-05 2024-09-16T13:29:14.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.632538+0000", 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8330000000000002e-05 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7024 4.1 4:876f6a7b:::rbd_data.11a976a8123d.000000000000000b:7 [sparse-read 2031616~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.757803598999999, 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015178159, 2024-09-16T13:29:14.150 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7024 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383041+0000", 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1899999999999998e-07 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383042+0000", 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.57e-06 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383047+0000", 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8500000000000002e-06 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.383050+0000", 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398103+0000", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052963000000001 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398151+0000", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8863999999999998e-05 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.398219+0000", 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7291000000000003e-05 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.154 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.430505377999999, 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.155 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:14.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:14.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.420966266999997, 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:14.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25661 4.d 4:b5278be0:::rbd_data.11a926bfcc53.0000000000000004:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.510265089000001, 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022042392000000001, 2024-09-16T13:29:14.165 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25661 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630579+0000", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630580+0000", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.37e-06 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630587+0000", 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.477e-06 2024-09-16T13:29:14.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630597+0000", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0473e-05 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630619+0000", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2050000000000001e-05 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630649+0000", 2024-09-16T13:29:14.168 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9513999999999999e-05 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.630704+0000", 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4753999999999998e-05 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.631810+0000", 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011062489999999999 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652526+0000", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020715588 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652559+0000", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3859999999999998e-05 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652594+0000", 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4749000000000002e-05 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652621+0000", 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7308e-05 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25661 4.d e48/32 ondisk, result = 0)", 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.505683285, 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017464550999999998, 2024-09-16T13:29:14.171 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635161+0000", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635162+0000", 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3200000000000004e-07 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635165+0000", 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5449999999999999e-06 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635178+0000", 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2461e-05 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.635182+0000", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.403e-06 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652548+0000", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017365642000000001 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652555+0000", 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.652625+0000", 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0648000000000005e-05 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.454509531999999, 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:14.175 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:14.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.445459683999999, 2024-09-16T13:29:14.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:29:14.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:29:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.744098081000001, 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:14.185 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:14.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.733451032000001, 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:14.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.195 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.066389450999999, 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:14.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:14.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38134 4.1b e59/32 ondisk, result = 0)", 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.056593093, 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036326266000000003, 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084252+0000", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1039999999999999e-06 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084256+0000", 2024-09-16T13:29:14.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5820000000000002e-06 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084270+0000", 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3752e-05 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084276+0000", 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6449999999999999e-06 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120563+0000", 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036286410999999998 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120571+0000", 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2139999999999996e-06 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120577+0000", 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.205 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50477 4.1 4:824a833b:::rbd_data.11a9b5ab4b7f.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.8691277670000002, 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016452580000000001, 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.206 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50477 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271716+0000", 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.271718+0000", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4440000000000001e-06 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272533+0000", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081477100000000005 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272547+0000", 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4780000000000001e-05 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272554+0000", 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4389999999999997e-06 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272600+0000", 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6511999999999998e-05 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272671+0000", 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0494000000000006e-05 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.273230+0000", 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055887199999999997 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288080+0000", 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014850070999999999 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288086+0000", 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2400000000000004e-06 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288123+0000", 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7302999999999998e-05 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288169+0000", 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5654e-05 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50478 4.1 4:824a833b:::rbd_data.11a9b5ab4b7f.0000000000000005:head [delete] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.8682763649999998, 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015740127, 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.212 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50478 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272568+0000", 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272569+0000", 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4199999999999995e-07 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272571+0000", 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8430000000000001e-06 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272578+0000", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8870000000000003e-06 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.272583+0000", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2479999999999998e-06 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279404+0000", 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068219409999999998 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279508+0000", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010376499999999999 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.279706+0000", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000198032 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288205+0000", 2024-09-16T13:29:14.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084983509999999995 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288219+0000", 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4511e-05 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288274+0000", 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4731000000000002e-05 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.288308+0000", 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4075999999999997e-05 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51816 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 737280~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8956543469999998, 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015237298, 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.218 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51816 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:14.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245190+0000", 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.48e-07 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245191+0000", 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2619999999999999e-06 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245196+0000", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4059999999999998e-06 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245198+0000", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2970000000000002e-06 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260130+0000", 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014932124999999999 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260176+0000", 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6211999999999997e-05 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260427+0000", 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025074800000000002 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51817 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 327680~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8956440360000002, 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015346831, 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51817 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245200+0000", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245201+0000", 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:29:14.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245202+0000", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.28e-06 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245205+0000", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1149999999999998e-06 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245207+0000", 2024-09-16T13:29:14.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.17e-06 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260443+0000", 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015236275000000001 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260490+0000", 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6252999999999999e-05 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260547+0000", 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7296000000000002e-05 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51818 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 786432~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8956350899999999, 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015474253, 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.227 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51818 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:14.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245209+0000", 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245211+0000", 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.646e-06 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245217+0000", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4370000000000004e-06 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245221+0000", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260574+0000", 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015352852 2024-09-16T13:29:14.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260623+0000", 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9394999999999997e-05 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260683+0000", 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0071000000000001e-05 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51819 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 540672~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8956194059999998, 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015585564999999999, 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51819 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.232 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245225+0000", 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245227+0000", 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1610000000000001e-06 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245234+0000", 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6189999999999996e-06 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245238+0000", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0300000000000004e-06 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260713+0000", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015474913 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260754+0000", 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1063999999999999e-05 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260810+0000", 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6388999999999998e-05 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51820 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 999424~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8956021920000001, 2024-09-16T13:29:14.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015898123, 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51820 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245242+0000", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.84e-07 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245244+0000", 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.883e-06 2024-09-16T13:29:14.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245250+0000", 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7470000000000002e-06 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245254+0000", 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.596e-06 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260837+0000", 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015583052999999999 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.260890+0000", 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3776999999999997e-05 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261140+0000", 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000249683 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.240 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:51821 4.1 4:83dc6e11:::rbd_data.11a9b5ab4b7f.0000000000000007:16 [read 589824~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.8955873270000003, 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016004555, 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51821 2024-09-16T13:29:14.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245257+0000", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1600000000000002e-07 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245259+0000", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245275+0000", 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5767999999999999e-05 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.245284+0000", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261159+0000", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015874632999999999 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261205+0000", 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6817999999999998e-05 2024-09-16T13:29:14.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:06.261261+0000", 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6039999999999999e-05 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.245 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:14.248 INFO:teuthology.orchestra.run.smithi076.stdout:2644 write 0xe402b9a thru 0xe40eaed (0xbf54 bytes) 2024-09-16T13:29:14.248 INFO:teuthology.orchestra.run.smithi076.stdout:2645 punch from 0x9139967 to 0x9146b90, (0xd229 bytes) 2024-09-16T13:29:14.248 INFO:teuthology.orchestra.run.smithi076.stdout:2646 read 0xb72dac6 thru 0xb734573 (0x6aae bytes) 2024-09-16T13:29:14.248 INFO:teuthology.orchestra.run.smithi076.stdout:2647 write 0x22234b3 thru 0x2231cb8 (0xe806 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2650 punch from 0x72dd559 to 0x72e490b, (0x73b2 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2651 write 0x5fc4fa7 thru 0x5fd29f7 (0xda51 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2652 writesame 0x5f88375 thru 0x5f8d14c data_size 0x8(0x4dd8 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2653 write 0x7934891 thru 0x793e470 (0x9be0 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2655 writesame 0x55e62a2 thru 0x55e78bf data_size 0x2(0x161e bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2656 read 0x74b775d thru 0x74c3223 (0xbac7 bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2657 writesame 0x95585f3 thru 0x955d89f data_size 0x33(0x52ad bytes) 2024-09-16T13:29:14.249 INFO:teuthology.orchestra.run.smithi076.stdout:2658 clone 23 order 18 su 16384 sc 2 2024-09-16T13:29:14.250 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:29:14.250 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:14.245+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:14.307 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone22 from 0xe40eaee (overlap 0x11d223a) to 0x0 2024-09-16T13:29:14.347 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:14.346+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:14.353 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.353 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:14.353 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:14.353 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:14.403 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-09-16T13:29:14.448 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:14.446+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:14.478 INFO:teuthology.orchestra.run.smithi076.stdout:2659 punch from 0x2380502 to 0x238dcaa, (0xd7a8 bytes) 2024-09-16T13:29:14.482 INFO:teuthology.orchestra.run.smithi076.stdout:2660 writesame 0x9cd8f28 thru 0x9cdb50b data_size 0x19(0x25e4 bytes) 2024-09-16T13:29:14.485 INFO:teuthology.orchestra.run.smithi076.stdout:2661 read 0xced9436 thru 0xcedde81 (0x4a4c bytes) 2024-09-16T13:29:14.486 INFO:teuthology.orchestra.run.smithi076.stdout:2662 punch from 0xc0abcc9 to 0xc0af4a0, (0x37d7 bytes) 2024-09-16T13:29:14.495 INFO:teuthology.orchestra.run.smithi076.stdout:2664 trunc from 0xe40eaee to 0x69097c1 2024-09-16T13:29:14.504 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.504 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:14.504 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:14.504 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:14.504 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.511 INFO:teuthology.orchestra.run.smithi076.stdout:2665 punch from 0x43b660c to 0x43c52ba, (0xecae bytes) 2024-09-16T13:29:14.511 INFO:teuthology.orchestra.run.smithi076.stdout:2666 write 0x286e182 thru 0x286ebd7 (0xa56 bytes) 2024-09-16T13:29:14.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:14.520 INFO:teuthology.orchestra.run.smithi076.stdout:2667 trunc from 0x69097c1 to 0x85a84c8 2024-09-16T13:29:14.526 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.533 INFO:teuthology.orchestra.run.smithi076.stdout:2671 writesame 0xa6b659b thru 0xa6c3da3 data_size 0xf(0xd809 bytes) 2024-09-16T13:29:14.536 INFO:teuthology.orchestra.run.smithi076.stdout:2672 read 0x389c273 thru 0x38aaca8 (0xea36 bytes) 2024-09-16T13:29:14.536 INFO:teuthology.orchestra.run.smithi076.stdout:2676 punch from 0x94de12d to 0x94ebae8, (0xd9bb bytes) 2024-09-16T13:29:14.536 INFO:teuthology.orchestra.run.smithi076.stdout:2681 write 0x1fa8f12 thru 0x1fb66ad (0xd79c bytes) 2024-09-16T13:29:14.542 INFO:teuthology.orchestra.run.smithi076.stdout:2682 write 0x92ebdd0 thru 0x92f9650 (0xd881 bytes) 2024-09-16T13:29:14.549 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:14.547+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:14.549 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.549 INFO:teuthology.orchestra.run.smithi076.stdout:2684 writesame 0x194afcd thru 0x1957cbe data_size 0x25(0xccf2 bytes) 2024-09-16T13:29:14.554 INFO:teuthology.orchestra.run.smithi076.stdout:2686 write 0x936efc8 thru 0x93712b2 (0x22eb bytes) 2024-09-16T13:29:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:2687 write 0x68a85f7 thru 0x68b373d (0xb147 bytes) 2024-09-16T13:29:14.566 INFO:teuthology.orchestra.run.smithi076.stdout:2688 read 0x8dfc6fd thru 0x8e0701c (0xa920 bytes) 2024-09-16T13:29:14.566 INFO:teuthology.orchestra.run.smithi076.stdout:2689 write 0x39901a1 thru 0x399089f (0x6ff bytes) 2024-09-16T13:29:14.572 INFO:teuthology.orchestra.run.smithi076.stdout:2690 read 0x7b17b39 thru 0x7b1989f (0x1d67 bytes) 2024-09-16T13:29:14.572 INFO:teuthology.orchestra.run.smithi076.stdout:2692 read 0x9d8d155 thru 0x9d93dac (0x6c58 bytes) 2024-09-16T13:29:14.572 INFO:teuthology.orchestra.run.smithi076.stdout:2693 read 0x24c2429 thru 0x24c921e (0x6df6 bytes) 2024-09-16T13:29:14.578 INFO:teuthology.orchestra.run.smithi076.stdout:2694 write 0xa9b0145 thru 0xa9b89e7 (0x88a3 bytes) 2024-09-16T13:29:14.581 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.592 INFO:teuthology.orchestra.run.smithi076.stdout:2698 trunc from 0xa9b89e8 to 0x1b407c9 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:2702 read 0x1932305 thru 0x1936015 (0x3d11 bytes) 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.612 INFO:teuthology.orchestra.run.smithi076.stdout:2705 read 0x166df8d thru 0x1675d70 (0x7de4 bytes) 2024-09-16T13:29:14.623 INFO:teuthology.orchestra.run.smithi076.stdout:2706 writesame 0x69665f0 thru 0x696dba7 data_size 0x4(0x75b8 bytes) 2024-09-16T13:29:14.635 INFO:teuthology.orchestra.run.smithi076.stdout:2707 writesame 0xbe1a363 thru 0xbe24318 data_size 0x1(0x9fb6 bytes) 2024-09-16T13:29:14.635 INFO:teuthology.orchestra.run.smithi076.stdout:2710 read 0x66693f8 thru 0x666a838 (0x1441 bytes) 2024-09-16T13:29:14.635 INFO:teuthology.orchestra.run.smithi076.stdout:2711 write 0x2c43ee9 thru 0x2c451ee (0x1306 bytes) 2024-09-16T13:29:14.638 INFO:teuthology.orchestra.run.smithi076.stdout:2712 read 0x3229da8 thru 0x322b48b (0x16e4 bytes) 2024-09-16T13:29:14.638 INFO:teuthology.orchestra.run.smithi076.stdout:2713 writesame 0x10f4e30 thru 0x10fdeec data_size 0x45(0x90bd bytes) 2024-09-16T13:29:14.643 INFO:teuthology.orchestra.run.smithi076.stdout:2717 read 0x1083207 thru 0x10888f1 (0x56eb bytes) 2024-09-16T13:29:14.643 INFO:teuthology.orchestra.run.smithi076.stdout:2722 write 0x6252186 thru 0x6252842 (0x6bd bytes) 2024-09-16T13:29:14.646 INFO:teuthology.orchestra.run.smithi076.stdout:2724 write 0x5b71299 thru 0x5b74b62 (0x38ca bytes) 2024-09-16T13:29:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:2725 read 0x86cfe21 thru 0x86ddeb4 (0xe094 bytes) 2024-09-16T13:29:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:2728 read 0xba8c551 thru 0xba93c60 (0x7710 bytes) 2024-09-16T13:29:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:2729 punch from 0xd3ee10 to 0xd4c217, (0xd407 bytes) 2024-09-16T13:29:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:2732 writesame 0x7a1c243 thru 0x7a2674c data_size 0x82(0xa50a bytes) 2024-09-16T13:29:14.650 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:14.648+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:14.652 INFO:teuthology.orchestra.run.smithi076.stdout:2733 writesame 0x43a7063 thru 0x43af3c1 data_size 0xd(0x835f bytes) 2024-09-16T13:29:14.656 INFO:teuthology.orchestra.run.smithi076.stdout:2734 writesame 0x9777b22 thru 0x97781dc data_size 0x1(0x6bb bytes) 2024-09-16T13:29:14.659 INFO:teuthology.orchestra.run.smithi076.stdout:2736 write 0xa8ff7e2 thru 0xa90db3f (0xe35e bytes) 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.018958865000002, 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:14.661 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:14.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:14.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:14.667 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.928428459999999, 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:14.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:14.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:14.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.673 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.901299289999997, 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:14.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.485537137999998, 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:14.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:14.682 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.415200855999998, 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:14.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:14.687 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.972079109999999, 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.688 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:14.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.961989843000001, 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:14.694 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.698 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.815201209000001, 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.699 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:14.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.704 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50161 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 565248~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4099053910000006, 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019286728999999999, 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.705 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50161 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248540+0000", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9499999999999998e-07 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248542+0000", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248547+0000", 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4870000000000002e-06 2024-09-16T13:29:14.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248551+0000", 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6500000000000002e-06 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267735+0000", 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019183892000000001 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267783+0000", 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7840999999999998e-05 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267826+0000", 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3695000000000001e-05 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50162 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 581632~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:14.709 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4098904680000004, 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019455525000000001, 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50162 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248555+0000", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248557+0000", 2024-09-16T13:29:14.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7850000000000001e-06 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248562+0000", 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8579999999999996e-06 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248566+0000", 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.207e-06 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267854+0000", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019288254000000001 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267938+0000", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3678999999999996e-05 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268010+0000", 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2416000000000006e-05 2024-09-16T13:29:14.713 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50163 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 598016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4098756170000009, 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019564818000000001, 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.714 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50163 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248570+0000", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248577+0000", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4320000000000002e-06 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248584+0000", 2024-09-16T13:29:14.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248587+0000", 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3790000000000002e-06 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268040+0000", 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019453427999999998 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268096+0000", 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5404e-05 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268134+0000", 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50164 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 614400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:14.718 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4098539419999998, 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019695615, 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50164 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248593+0000", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9300000000000002e-06 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248599+0000", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7950000000000004e-06 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248603+0000", 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268171+0000", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019568232000000001 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268243+0000", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1872000000000006e-05 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268287+0000", 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3300000000000002e-05 2024-09-16T13:29:14.722 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50165 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 630784~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4098377539999998, 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019831332, 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.723 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50165 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248608+0000", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248610+0000", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.807e-06 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248619+0000", 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4779999999999997e-06 2024-09-16T13:29:14.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248623+0000", 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0489999999999997e-06 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268313+0000", 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019690151999999999 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268379+0000", 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5792999999999998e-05 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268439+0000", 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537000000000003e-05 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50166 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:14.727 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4098179829999999, 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019843890999999999, 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50166 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:14.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.65e-07 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248629+0000", 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.854e-06 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248635+0000", 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248645+0000", 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7599999999999997e-06 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268466+0000", 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019821442000000002 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268471+0000", 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.775e-06 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.731 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50167 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4097963530000008, 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019857698999999999, 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50167 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248651+0000", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.181e-06 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248657+0000", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0939999999999996e-06 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248661+0000", 2024-09-16T13:29:14.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268499+0000", 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019837751000000001 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268506+0000", 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7440000000000007e-06 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50168 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.4097806399999993, 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019875693999999999, 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50168 2024-09-16T13:29:14.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248665+0000", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248667+0000", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.866e-06 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248672+0000", 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3410000000000002e-06 2024-09-16T13:29:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248675+0000", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.264e-06 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268536+0000", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019860932000000001 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268540+0000", 2024-09-16T13:29:14.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50928 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 311296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.3617817809999995, 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019281187000000002, 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.740 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50928 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296663+0000", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0600000000000001e-07 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296665+0000", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5069999999999999e-06 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296670+0000", 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9930000000000003e-06 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296673+0000", 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.151e-06 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315875+0000", 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019202 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315910+0000", 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5036000000000001e-05 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315944+0000", 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4193999999999998e-05 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50929 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 327680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.3617686550000005, 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019357040999999998, 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50929 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296677+0000", 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9900000000000002e-07 2024-09-16T13:29:14.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296678+0000", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.401e-06 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296683+0000", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296686+0000", 2024-09-16T13:29:14.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8949999999999998e-06 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315967+0000", 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019280701000000001 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316000+0000", 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3549e-05 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316033+0000", 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3272999999999998e-05 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50930 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 344064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.3617563120000007, 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019423292000000002, 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.749 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50930 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5499999999999999e-07 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296690+0000", 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.575e-06 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296696+0000", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3460000000000001e-06 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296704+0000", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6319999999999997e-06 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316046+0000", 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019341649999999998 2024-09-16T13:29:14.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316079+0000", 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2898999999999998e-05 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316112+0000", 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2935000000000002e-05 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6832768160000002, 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:14.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:14.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:14.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:14.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:14.760 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:14.762 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:14.760+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2737 punch from 0x98fc212 to 0x990b50f, (0xf2fd bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2739 punch from 0x6ad262c to 0x6ad2a46, (0x41a bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2740 write 0xd647e04 thru 0xd652a15 (0xac12 bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2741 write 0xa035aec thru 0xa03b66b (0x5b80 bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2742 write 0x6af8fcf thru 0x6afca84 (0x3ab6 bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2743 writesame 0xc53c270 thru 0xc5416bf data_size 0x4c(0x5450 bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2744 read 0x931a64b thru 0x9329f37 (0xf8ed bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2747 write 0x772ed20 thru 0x7738609 (0x98ea bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2748 write 0xd4d4826 thru 0xd4e1a58 (0xd233 bytes) 2024-09-16T13:29:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:2749 writesame 0x85fdfa4 thru 0x86087ef data_size 0x2(0xa84c bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2750 writesame 0xb77f7a7 thru 0xb78279f data_size 0x1(0x2ff9 bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2752 read 0xc2a9774 thru 0xc2b238e (0x8c1b bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2753 punch from 0x7275b98 to 0x7276445, (0x8ad bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2755 write 0x3e2872c thru 0x3e307cb (0x80a0 bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2756 read 0x733ba6e thru 0x733f9ed (0x3f80 bytes) 2024-09-16T13:29:14.765 INFO:teuthology.orchestra.run.smithi076.stdout:2758 writesame 0x2606035 thru 0x260c00e data_size 0x1(0x5fda bytes) 2024-09-16T13:29:14.771 INFO:teuthology.orchestra.run.smithi076.stdout:2761 punch from 0x9a56f12 to 0x9a63752, (0xc840 bytes) 2024-09-16T13:29:14.779 INFO:teuthology.orchestra.run.smithi076.stdout:2762 trunc from 0xd652a16 to 0x66c67c1 2024-09-16T13:29:14.794 INFO:teuthology.orchestra.run.smithi076.stdout:2763 trunc from 0x66c67c1 to 0x5e1623e 2024-09-16T13:29:14.802 INFO:teuthology.orchestra.run.smithi076.stdout:2764 read 0x2306e14 thru 0x231058c (0x9779 bytes) 2024-09-16T13:29:14.807 INFO:teuthology.orchestra.run.smithi076.stdout:2765 write 0x5e92545 thru 0x5e9f81c (0xd2d8 bytes) 2024-09-16T13:29:14.820 INFO:teuthology.orchestra.run.smithi076.stdout:2766 trunc from 0x5e9f81d to 0xe164de2 2024-09-16T13:29:14.825 INFO:teuthology.orchestra.run.smithi076.stdout:2767 writesame 0xad57eb4 thru 0xad58db3 data_size 0x10(0xf00 bytes) 2024-09-16T13:29:14.843 INFO:teuthology.orchestra.run.smithi076.stdout:2768 trunc from 0xe164de2 to 0x22d9102 2024-09-16T13:29:14.862 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:14.860+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:14.866 INFO:teuthology.orchestra.run.smithi076.stdout:2769 write 0x7cd3715 thru 0x7cdfd7e (0xc66a bytes) 2024-09-16T13:29:14.871 INFO:teuthology.orchestra.run.smithi076.stdout:2770 trunc from 0x7cdfd7f to 0x8f3b142 2024-09-16T13:29:14.876 INFO:teuthology.orchestra.run.smithi076.stdout:2773 writesame 0x2910f38 thru 0x29124b1 data_size 0x1(0x157a bytes) 2024-09-16T13:29:14.881 INFO:teuthology.orchestra.run.smithi076.stdout:2774 read 0x611d497 thru 0x6123202 (0x5d6c bytes) 2024-09-16T13:29:14.892 INFO:teuthology.orchestra.run.smithi076.stdout:2775 write 0xd906ce2 thru 0xd90d7ca (0x6ae9 bytes) 2024-09-16T13:29:14.895 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.896 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.896 INFO:teuthology.orchestra.run.smithi076.stdout:2780 writesame 0x41cac4f thru 0x41d476d data_size 0x5d(0x9b1f bytes) 2024-09-16T13:29:14.899 INFO:teuthology.orchestra.run.smithi076.stdout:2781 write 0xb0e89b1 thru 0xb0ed5b5 (0x4c05 bytes) 2024-09-16T13:29:14.902 INFO:teuthology.orchestra.run.smithi076.stdout:2783 writesame 0x6d36067 thru 0x6d38faf data_size 0x5(0x2f49 bytes) 2024-09-16T13:29:14.906 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:14.907 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:14.907 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:14.907 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:14.908 INFO:teuthology.orchestra.run.smithi076.stdout:2784 write 0x235d7f3 thru 0x2363466 (0x5c74 bytes) 2024-09-16T13:29:14.913 INFO:teuthology.orchestra.run.smithi076.stdout:2785 punch from 0xb6c2517 to 0xb6c674f, (0x4238 bytes) 2024-09-16T13:29:14.913 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.913 INFO:teuthology.orchestra.run.smithi076.stdout:2787 write 0x3622aa7 thru 0x36265c6 (0x3b20 bytes) 2024-09-16T13:29:14.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:14.921 INFO:teuthology.orchestra.run.smithi076.stdout:2788 writesame 0x7e053f6 thru 0x7e0f843 data_size 0x1(0xa44e bytes) 2024-09-16T13:29:14.960 INFO:teuthology.orchestra.run.smithi076.stdout:2789 read 0xebf84f thru 0xec5ea2 (0x6654 bytes) 2024-09-16T13:29:14.960 INFO:teuthology.orchestra.run.smithi076.stdout:2790 write 0x28b8d83 thru 0x28c682f (0xdaad bytes) 2024-09-16T13:29:14.962 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:14.960+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:14.967 INFO:teuthology.orchestra.run.smithi076.stdout:2792 read 0x7b802cd thru 0x7b8573d (0x5471 bytes) 2024-09-16T13:29:14.967 INFO:teuthology.orchestra.run.smithi076.stdout:2794 write 0x1c8cee2 thru 0x1c98b48 (0xbc67 bytes) 2024-09-16T13:29:14.973 INFO:teuthology.orchestra.run.smithi076.stdout:2795 punch from 0xa9e602f to 0xa9f2b9d, (0xcb6e bytes) 2024-09-16T13:29:14.973 INFO:teuthology.orchestra.run.smithi076.stdout:2796 read 0x5ded687 thru 0x5df2ea1 (0x581b bytes) 2024-09-16T13:29:14.973 INFO:teuthology.orchestra.run.smithi076.stdout:2797 punch from 0x930756f to 0x930a13d, (0x2bce bytes) 2024-09-16T13:29:14.973 INFO:teuthology.orchestra.run.smithi076.stdout:2798 read 0x4c0b162 thru 0x4c100d5 (0x4f74 bytes) 2024-09-16T13:29:14.974 INFO:teuthology.orchestra.run.smithi076.stdout:2799 read 0xa36a1e4 thru 0xa377473 (0xd290 bytes) 2024-09-16T13:29:14.974 INFO:teuthology.orchestra.run.smithi076.stdout:2801 read 0x9a8bdb7 thru 0x9a931e1 (0x742b bytes) 2024-09-16T13:29:14.974 INFO:teuthology.orchestra.run.smithi076.stdout:2802 read 0xb4ad40c thru 0xb4bbdf5 (0xe9ea bytes) 2024-09-16T13:29:14.974 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:14.974 INFO:teuthology.orchestra.run.smithi076.stdout:2804 punch from 0x88f334 to 0x89a0f1, (0xadbd bytes) 2024-09-16T13:29:14.979 INFO:teuthology.orchestra.run.smithi076.stdout:2806 trunc from 0xd90d7cb to 0x8a8a183 2024-09-16T13:29:14.988 INFO:teuthology.orchestra.run.smithi076.stdout:2808 read 0x30adaab thru 0x30b498d (0x6ee3 bytes) 2024-09-16T13:29:14.995 INFO:teuthology.orchestra.run.smithi076.stdout:2809 trunc from 0x8a8a183 to 0xe0a2e9d 2024-09-16T13:29:15.000 INFO:teuthology.orchestra.run.smithi076.stdout:2810 punch from 0xbecb170 to 0xbecf48c, (0x431c bytes) 2024-09-16T13:29:15.000 INFO:teuthology.orchestra.run.smithi076.stdout:2811 write 0x4fa009d thru 0x4fa4a09 (0x496d bytes) 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:2813 punch from 0xccb8e3b to 0xccbe94c, (0x5b11 bytes) 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:2814 read 0xc7db4dd thru 0xc7e51a6 (0x9cca bytes) 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:2816 read 0x7e7d435 thru 0x7e860c9 (0x8c95 bytes) 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:2817 read 0xa98f701 thru 0xa99d51b (0xde1b bytes) 2024-09-16T13:29:15.007 INFO:teuthology.orchestra.run.smithi076.stdout:2818 writesame 0xb34053f thru 0xb34eac7 data_size 0x3(0xe589 bytes) 2024-09-16T13:29:15.016 INFO:teuthology.orchestra.run.smithi076.stdout:2819 write 0x3dab74 thru 0x3df749 (0x4bd6 bytes) 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:15.024 INFO:teuthology.orchestra.run.smithi076.stdout:2820 write 0x45a0b3b thru 0x45a9f8d (0x9453 bytes) 2024-09-16T13:29:15.028 INFO:teuthology.orchestra.run.smithi076.stdout:2821 write 0x75ffb8b thru 0x760e810 (0xec86 bytes) 2024-09-16T13:29:15.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:15.035 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.036 INFO:teuthology.orchestra.run.smithi076.stdout:2823 write 0xcfca3e6 thru 0xcfce0c8 (0x3ce3 bytes) 2024-09-16T13:29:15.041 INFO:teuthology.orchestra.run.smithi076.stdout:2824 write 0xd2b4b3 thru 0xd3677a (0xb2c8 bytes) 2024-09-16T13:29:15.046 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.063 INFO:teuthology.orchestra.run.smithi076.stdout:2828 trunc from 0xe0a2e9d to 0x35b8e 2024-09-16T13:29:15.063 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:15.061+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:15.098 INFO:teuthology.orchestra.run.smithi076.stdout:2830 write 0x6698e59 thru 0x669ce7d (0x4025 bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2832 writesame 0xc1c403d thru 0xc1ca41f data_size 0x7(0x63e3 bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2833 read 0x78499fd thru 0x784f7f7 (0x5dfb bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2836 read 0x9362aee thru 0x936f026 (0xc539 bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2837 punch from 0x33d538d to 0x33de012, (0x8c85 bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2840 read 0x1d6a7b0 thru 0x1d78bcf (0xe420 bytes) 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.114 INFO:teuthology.orchestra.run.smithi076.stdout:2844 writesame 0x4f694a3 thru 0x4f7217d data_size 0x1(0x8cdb bytes) 2024-09-16T13:29:15.149 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.161 INFO:teuthology.orchestra.run.smithi076.stdout:2847 trunc from 0xc1ca420 to 0x24c0e47 2024-09-16T13:29:15.164 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:15.161+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:15.170 INFO:teuthology.orchestra.run.smithi076.stdout:2848 read 0x1388667 thru 0x138a0e0 (0x1a7a bytes) 2024-09-16T13:29:15.177 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:15.177 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:15.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.487598093000003, 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.178 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:15.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.478026614999997, 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:15.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:15.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:15.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.194 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.630630203000003, 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:29:15.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:29:15.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:15.199 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.630617933000003, 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:29:15.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:15.204 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.630607087999998, 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:15.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:29:15.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:29:15.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.630597674000001, 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:15.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:15.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884358540999997, 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:29:15.213 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:15.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.218 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884350196, 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:29:15.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.224 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884339502000003, 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:15.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:29:15.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884330513999998, 2024-09-16T13:29:15.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.233 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:15.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:15.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884322773000001, 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.239 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:15.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.243 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884314214, 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:15.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:15.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.248 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884305337000001, 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884291472000001, 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.253 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:15.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:15.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:15.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884283005, 2024-09-16T13:29:15.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.261 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:15.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.884274202, 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.269 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:15.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.260804144000002, 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:15.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:15.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.247575463, 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:15.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.733749274000001, 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:29:15.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:29:15.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.716693396, 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.291 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:29:15.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.295 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:15.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2849 trunc from 0x24c0e47 to 0x9d79657 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2850 read 0x121ab2e thru 0x1228846 (0xdd19 bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2851 writesame 0xe5cb4da thru 0xe5d43ef data_size 0x1e(0x8f16 bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2852 write 0x3973097 thru 0x397735d (0x42c7 bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2853 punch from 0x2969e80 to 0x296d04e, (0x31ce bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2854 trunc from 0xe5d43f0 to 0xaee83e8 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2856 write 0xde3799c thru 0xde3b21d (0x3882 bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2857 write 0x705c4f2 thru 0x70676dd (0xb1ec bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2858 punch from 0x4ca6b60 to 0x4cb3584, (0xca24 bytes) 2024-09-16T13:29:15.299 INFO:teuthology.orchestra.run.smithi076.stdout:2860 writesame 0xcca0ffa thru 0xcca20df data_size 0x15(0x10e6 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2863 read 0x53d1b08 thru 0x53d8e9b (0x7394 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2864 writesame 0x7cb619b thru 0x7cbbeff data_size 0x1(0x5d65 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2865 read 0xc7d209f thru 0xc7d8b59 (0x6abb bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2867 write 0xb55acf7 thru 0xb562249 (0x7553 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2869 punch from 0x770917 to 0x7777a0, (0x6e89 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2870 write 0x9df289 thru 0x9e95c7 (0xa33f bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2871 write 0x9a78d17 thru 0x9a7f0eb (0x63d5 bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2872 punch from 0x2a583c4 to 0x2a66571, (0xe1ad bytes) 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2874 trunc from 0xde3b21e to 0x1476aed 2024-09-16T13:29:15.300 INFO:teuthology.orchestra.run.smithi076.stdout:2875 punch from 0x304a8c to 0x30d0a0, (0x8614 bytes) 2024-09-16T13:29:15.300 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:15.295+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:15.307 INFO:teuthology.orchestra.run.smithi076.stdout:2876 write 0xe9c1aaf thru 0xe9cd2cd (0xb81f bytes) 2024-09-16T13:29:15.324 INFO:teuthology.orchestra.run.smithi076.stdout:2877 trunc from 0xe9cd2ce to 0x5bcbe46 2024-09-16T13:29:15.334 INFO:teuthology.orchestra.run.smithi076.stdout:2878 read 0x188e101 thru 0x1890a6b (0x296b bytes) 2024-09-16T13:29:15.334 INFO:teuthology.orchestra.run.smithi076.stdout:2880 writesame 0x1c5d096 thru 0x1c66d94 data_size 0x1(0x9cff bytes) 2024-09-16T13:29:15.381 INFO:teuthology.orchestra.run.smithi076.stdout:2881 writesame 0xc77175 thru 0xc78c85 data_size 0xd(0x1b11 bytes) 2024-09-16T13:29:15.384 INFO:teuthology.orchestra.run.smithi076.stdout:2882 writesame 0x32f4fec thru 0x32fe755 data_size 0x1(0x976a bytes) 2024-09-16T13:29:15.397 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:15.395+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:15.425 INFO:teuthology.orchestra.run.smithi076.stdout:2883 read 0x2f395c3 thru 0x2f3af73 (0x19b1 bytes) 2024-09-16T13:29:15.425 INFO:teuthology.orchestra.run.smithi076.stdout:2886 read 0x3d6e1cf thru 0x3d7c7ce (0xe600 bytes) 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:2889 read 0x412cd1e thru 0x412ebd2 (0x1eb5 bytes) 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:2892 read 0x2a6609e thru 0x2a6ff34 (0x9e97 bytes) 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:2895 read 0x4648ffd thru 0x46510db (0x80df bytes) 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:2896 read 0x2ce29b1 thru 0x2ceeed2 (0xc522 bytes) 2024-09-16T13:29:15.426 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.436 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:15.437 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:15.437 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:15.437 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:15.439 INFO:teuthology.orchestra.run.smithi076.stdout:2899 writesame 0xc7588a5 thru 0xc7671f3 data_size 0x2b(0xe94f bytes) 2024-09-16T13:29:15.439 INFO:teuthology.orchestra.run.smithi076.stdout:2900 read 0xa33262e thru 0xa33d6e7 (0xb0ba bytes) 2024-09-16T13:29:15.439 INFO:teuthology.orchestra.run.smithi076.stdout:2902 read 0xad4f08b thru 0xad57345 (0x82bb bytes) 2024-09-16T13:29:15.439 INFO:teuthology.orchestra.run.smithi076.stdout:2904 write 0xb0650b1 thru 0xb0705ab (0xb4fb bytes) 2024-09-16T13:29:15.445 INFO:teuthology.orchestra.run.smithi076.stdout:2905 writesame 0xaa187f5 thru 0xaa26618 data_size 0x2a(0xde24 bytes) 2024-09-16T13:29:15.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:15.451 INFO:teuthology.orchestra.run.smithi076.stdout:2906 read 0x2585190 thru 0x2586bef (0x1a60 bytes) 2024-09-16T13:29:15.459 INFO:teuthology.orchestra.run.smithi076.stdout:2907 write 0xe2e0442 thru 0xe2e4171 (0x3d30 bytes) 2024-09-16T13:29:15.466 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.466 INFO:teuthology.orchestra.run.smithi076.stdout:2909 writesame 0x661865 thru 0x66197f data_size 0x1(0x11b bytes) 2024-09-16T13:29:15.469 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:15.469 INFO:teuthology.orchestra.run.smithi076.stdout:2912 write 0x43af92 thru 0x44894f (0xd9be bytes) 2024-09-16T13:29:15.474 INFO:teuthology.orchestra.run.smithi076.stdout:2913 read 0xade9e8e thru 0xadf9842 (0xf9b5 bytes) 2024-09-16T13:29:15.474 INFO:teuthology.orchestra.run.smithi076.stdout:2915 punch from 0x434de2f to 0x4359acd, (0xbc9e bytes) 2024-09-16T13:29:15.487 INFO:teuthology.orchestra.run.smithi076.stdout:2916 trunc from 0xe2e4172 to 0x29fe597 2024-09-16T13:29:15.498 INFO:teuthology.orchestra.run.smithi076.stdout:2919 write 0xd93caa thru 0xd95aa1 (0x1df8 bytes) 2024-09-16T13:29:15.498 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:15.495+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:15.508 INFO:teuthology.orchestra.run.smithi076.stdout:2920 writesame 0x4085100 thru 0x408af81 data_size 0x1(0x5e82 bytes) 2024-09-16T13:29:15.508 INFO:teuthology.orchestra.run.smithi076.stdout:2921 clone 24 order 18 su 32768 sc 8 2024-09-16T13:29:15.547 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:15.548 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:15.557 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:15.599 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:15.597+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:15.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 70.404807019000003, 2024-09-16T13:29:15.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:15.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:15.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.703456822000007, 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:29:15.704 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:29:15.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:29:15.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.703334425999998, 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:15.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.716 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.703222038000007, 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076667280000000003, 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.717 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 43 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990786+0000", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2399999999999998e-07 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990790+0000", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990796+0000", 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4899999999999997e-06 2024-09-16T13:29:15.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990802+0000", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993164+0000", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023615860000000002 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993214+0000", 2024-09-16T13:29:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9923999999999998e-05 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993295+0000", 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1157999999999998e-05 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996367+0000", 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003072459 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998406+0000", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020384140000000001 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998433+0000", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6922000000000001e-05 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998452+0000", 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9466e-05 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.722 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.703159224000004, 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.723 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:15.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:15.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:15.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702996611000003, 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.729 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702975563999999, 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.735 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:15.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:15.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702888083999994, 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.742 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:15.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:15.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702865575000004, 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:15.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:15.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:15.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:15.755 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702819247999997, 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:15.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:15.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:15.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702652490999995, 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:15.765 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:15.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.702637726000006, 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.772 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:15.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:15.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.777 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.357315408999995, 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.778 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:15.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:15.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:15.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.357277730999996, 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:15.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:15.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.357254917999995, 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:15.791 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.357169705000004, 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:15.798 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:15.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.05059078, 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.809 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.024682546000001, 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:15.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:15.815 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.955917391, 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.454374574, 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:15.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:15.825 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:15.825 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:15.825 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:15.826 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:15.827 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:15.827 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:15.827 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:15.824+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:15.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:15.828 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:15.829 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:15.840 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:15.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:15.857 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:15.857 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:15.872 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:15.872 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:15.887 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:15.888 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:15.903 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:15.903 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:15.928 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:15.928 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:15.929 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:15.927+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:15.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:16.028 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:16.027+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:16.076 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.077 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.085 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:16.171 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:16.169+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:16.224 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.224 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.229 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:16.226+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:16.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:16.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:29:16.325 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone23 from 0x408af82 (overlap 0x35b8e) to 0x1adc7 2024-09-16T13:29:16.330 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:16.328+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:16.384 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.384 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:16.409 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-09-16T13:29:16.427 INFO:teuthology.orchestra.run.smithi076.stdout:2922 write 0xbd39cc9 thru 0xbd48fdb (0xf313 bytes) 2024-09-16T13:29:16.431 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:16.429+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:16.432 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.432 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.440 INFO:teuthology.orchestra.run.smithi076.stdout:2925 trunc from 0xbd48fdc to 0x5154634 2024-09-16T13:29:16.449 INFO:teuthology.orchestra.run.smithi076.stdout:2926 punch from 0x2954ebd to 0x29638a6, (0xe9e9 bytes) 2024-09-16T13:29:16.449 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.449 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.449 INFO:teuthology.orchestra.run.smithi076.stdout:2930 read 0x11d5e52 thru 0x11dd814 (0x79c3 bytes) 2024-09-16T13:29:16.449 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.450 INFO:teuthology.orchestra.run.smithi076.stdout:2932 read 0x50410d6 thru 0x504bbb3 (0xaade bytes) 2024-09-16T13:29:16.450 INFO:teuthology.orchestra.run.smithi076.stdout:2933 read 0x4729848 thru 0x473885b (0xf014 bytes) 2024-09-16T13:29:16.453 INFO:teuthology.orchestra.run.smithi076.stdout:2935 trunc from 0x5154634 to 0x87b9fc7 2024-09-16T13:29:16.458 INFO:teuthology.orchestra.run.smithi076.stdout:2937 read 0x1a1c3dc thru 0x1a21f61 (0x5b86 bytes) 2024-09-16T13:29:16.458 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.468 INFO:teuthology.orchestra.run.smithi076.stdout:2941 write 0xd6643ff thru 0xd667ed8 (0x3ada bytes) 2024-09-16T13:29:16.471 INFO:teuthology.orchestra.run.smithi076.stdout:2942 punch from 0x2a8a3dc to 0x2a92eee, (0x8b12 bytes) 2024-09-16T13:29:16.471 INFO:teuthology.orchestra.run.smithi076.stdout:2943 writesame 0x574688e thru 0x574e587 data_size 0x11(0x7cfa bytes) 2024-09-16T13:29:16.478 INFO:teuthology.orchestra.run.smithi076.stdout:2944 punch from 0x2803ad3 to 0x28072dc, (0x3809 bytes) 2024-09-16T13:29:16.478 INFO:teuthology.orchestra.run.smithi076.stdout:2945 punch from 0x1564650 to 0x156cb10, (0x84c0 bytes) 2024-09-16T13:29:16.478 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.478 INFO:teuthology.orchestra.run.smithi076.stdout:2947 read 0x7091df8 thru 0x709f1f5 (0xd3fe bytes) 2024-09-16T13:29:16.480 INFO:teuthology.orchestra.run.smithi076.stdout:2948 trunc from 0xd667ed9 to 0xbd13e50 2024-09-16T13:29:16.490 INFO:teuthology.orchestra.run.smithi076.stdout:2949 write 0x181da63 thru 0x1824f4c (0x74ea bytes) 2024-09-16T13:29:16.497 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.497 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.497 INFO:teuthology.orchestra.run.smithi076.stdout:2953 write 0x9bbfd7b thru 0x9bcea58 (0xecde bytes) 2024-09-16T13:29:16.506 INFO:teuthology.orchestra.run.smithi076.stdout:2954 writesame 0x20d42aa thru 0x20da3a2 data_size 0x19(0x60f9 bytes) 2024-09-16T13:29:16.511 INFO:teuthology.orchestra.run.smithi076.stdout:2955 read 0x5254e67 thru 0x525fbf4 (0xad8e bytes) 2024-09-16T13:29:16.511 INFO:teuthology.orchestra.run.smithi076.stdout:2956 write 0x5d7c759 thru 0x5d8769e (0xaf46 bytes) 2024-09-16T13:29:16.518 INFO:teuthology.orchestra.run.smithi076.stdout:2957 punch from 0x6d5af1c to 0x6d5b743, (0x827 bytes) 2024-09-16T13:29:16.518 INFO:teuthology.orchestra.run.smithi076.stdout:2958 punch from 0x532742 to 0x5330fc, (0x9ba bytes) 2024-09-16T13:29:16.518 INFO:teuthology.orchestra.run.smithi076.stdout:2960 read 0xb4662ce thru 0xb472f92 (0xccc5 bytes) 2024-09-16T13:29:16.518 INFO:teuthology.orchestra.run.smithi076.stdout:2961 writesame 0x36679cb thru 0x366bf86 data_size 0x1(0x45bc bytes) 2024-09-16T13:29:16.526 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.526 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.531 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:16.530+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:16.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:16.546 INFO:teuthology.orchestra.run.smithi076.stdout:2963 read 0x3c84a66 thru 0x3c947e5 (0xfd80 bytes) 2024-09-16T13:29:16.547 INFO:teuthology.orchestra.run.smithi076.stdout:2964 punch from 0x6282b4c to 0x6287ab1, (0x4f65 bytes) 2024-09-16T13:29:16.547 INFO:teuthology.orchestra.run.smithi076.stdout:2966 writesame 0x34fef2c thru 0x3509360 data_size 0x1(0xa435 bytes) 2024-09-16T13:29:16.616 INFO:teuthology.orchestra.run.smithi076.stdout:2968 write 0xb45b2d3 thru 0xb45dfdc (0x2d0a bytes) 2024-09-16T13:29:16.619 INFO:teuthology.orchestra.run.smithi076.stdout:2970 writesame 0x4a429d6 thru 0x4a515b8 data_size 0x1(0xebe3 bytes) 2024-09-16T13:29:16.633 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:16.630+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:16.651 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.651 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:16.703 INFO:teuthology.orchestra.run.smithi076.stdout:2971 write 0x1810fe1 thru 0x1811db4 (0xdd4 bytes) 2024-09-16T13:29:16.714 INFO:teuthology.orchestra.run.smithi076.stdout:2972 trunc from 0xbd13e50 to 0x4c25c71 2024-09-16T13:29:16.734 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:16.731+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:16.737 INFO:teuthology.orchestra.run.smithi076.stdout:2973 writesame 0xabe7779 thru 0xabe9814 data_size 0x2(0x209c bytes) 2024-09-16T13:29:16.737 INFO:teuthology.orchestra.run.smithi076.stdout:2974 read 0x7905bf6 thru 0x79144ed (0xe8f8 bytes) 2024-09-16T13:29:16.737 INFO:teuthology.orchestra.run.smithi076.stdout:2975 punch from 0x1ec109c to 0x1ecf216, (0xe17a bytes) 2024-09-16T13:29:16.737 INFO:teuthology.orchestra.run.smithi076.stdout:2976 punch from 0x78de592 to 0x78e2d74, (0x47e2 bytes) 2024-09-16T13:29:16.737 INFO:teuthology.orchestra.run.smithi076.stdout:2977 write 0x61e7f34 thru 0x61e8881 (0x94e bytes) 2024-09-16T13:29:16.747 INFO:teuthology.orchestra.run.smithi076.stdout:2978 writesame 0xd0c92bb thru 0xd0d5bbe data_size 0x1f(0xc904 bytes) 2024-09-16T13:29:16.748 INFO:teuthology.orchestra.run.smithi076.stdout:2979 punch from 0xb283d58 to 0xb293523, (0xf7cb bytes) 2024-09-16T13:29:16.748 INFO:teuthology.orchestra.run.smithi076.stdout:2980 punch from 0x9efa330 to 0x9eff516, (0x51e6 bytes) 2024-09-16T13:29:16.748 INFO:teuthology.orchestra.run.smithi076.stdout:2982 read 0xc481c38 thru 0xc483b1e (0x1ee7 bytes) 2024-09-16T13:29:16.748 INFO:teuthology.orchestra.run.smithi076.stdout:2983 write 0x9009f2e thru 0x900ba62 (0x1b35 bytes) 2024-09-16T13:29:16.750 INFO:teuthology.orchestra.run.smithi076.stdout:2984 write 0x2f5dd1b thru 0x2f6c67a (0xe960 bytes) 2024-09-16T13:29:16.756 INFO:teuthology.orchestra.run.smithi076.stdout:2986 writesame 0x23cb00e thru 0x23d9132 data_size 0x1(0xe125 bytes) 2024-09-16T13:29:16.776 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.777 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:16.835 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:16.833+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:16.840 INFO:teuthology.orchestra.run.smithi076.stdout:2987 trunc from 0xd0d5bbf to 0x5407625 2024-09-16T13:29:16.849 INFO:teuthology.orchestra.run.smithi076.stdout:2988 punch from 0x17b8869 to 0x17bd7e0, (0x4f77 bytes) 2024-09-16T13:29:16.864 INFO:teuthology.orchestra.run.smithi076.stdout:2990 writesame 0xd9315ef thru 0xd9350ec data_size 0x9(0x3afe bytes) 2024-09-16T13:29:16.864 INFO:teuthology.orchestra.run.smithi076.stdout:2992 write 0x93b929 thru 0x943906 (0x7fde bytes) 2024-09-16T13:29:16.869 INFO:teuthology.orchestra.run.smithi076.stdout:2993 write 0xb1a8d77 thru 0xb1b055f (0x77e9 bytes) 2024-09-16T13:29:16.873 INFO:teuthology.orchestra.run.smithi076.stdout:2996 read 0x8940161 thru 0x89408dc (0x77c bytes) 2024-09-16T13:29:16.873 INFO:teuthology.orchestra.run.smithi076.stdout:2997 write 0xadc313 thru 0xae02c9 (0x3fb7 bytes) 2024-09-16T13:29:16.878 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.878 INFO:teuthology.orchestra.run.smithi076.stdout:3001 read 0x266e22b thru 0x26766e5 (0x84bb bytes) 2024-09-16T13:29:16.878 INFO:teuthology.orchestra.run.smithi076.stdout:3002 read 0xd4283dc thru 0xd434fd4 (0xcbf9 bytes) 2024-09-16T13:29:16.880 INFO:teuthology.orchestra.run.smithi076.stdout:3004 trunc from 0xd9350ed to 0xbcd8b47 2024-09-16T13:29:16.885 INFO:teuthology.orchestra.run.smithi076.stdout:3005 read 0xb45268a thru 0xb458b48 (0x64bf bytes) 2024-09-16T13:29:16.885 INFO:teuthology.orchestra.run.smithi076.stdout:3007 punch from 0x5adcfb1 to 0x5ae1196, (0x41e5 bytes) 2024-09-16T13:29:16.888 INFO:teuthology.orchestra.run.smithi076.stdout:3008 trunc from 0xbcd8b47 to 0xdd2ff9f 2024-09-16T13:29:16.893 INFO:teuthology.orchestra.run.smithi076.stdout:3009 trunc from 0xdd2ff9f to 0xe432a11 2024-09-16T13:29:16.898 INFO:teuthology.orchestra.run.smithi076.stdout:3010 writesame 0x3c7889b thru 0x3c7c786 data_size 0x2(0x3eec bytes) 2024-09-16T13:29:16.901 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:16.901 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:16.906 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.906 INFO:teuthology.orchestra.run.smithi076.stdout:3013 write 0x586e232 thru 0x58718d0 (0x369f bytes) 2024-09-16T13:29:16.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:16.914 INFO:teuthology.orchestra.run.smithi076.stdout:3014 punch from 0x3708489 to 0x371326c, (0xade3 bytes) 2024-09-16T13:29:16.914 INFO:teuthology.orchestra.run.smithi076.stdout:3017 read 0x945eaa7 thru 0x945f615 (0xb6f bytes) 2024-09-16T13:29:16.914 INFO:teuthology.orchestra.run.smithi076.stdout:3019 read 0x97d3df7 thru 0x97d4621 (0x82b bytes) 2024-09-16T13:29:16.914 INFO:teuthology.orchestra.run.smithi076.stdout:3021 write 0xe020c5a thru 0xe030a5d (0xfe04 bytes) 2024-09-16T13:29:16.920 INFO:teuthology.orchestra.run.smithi076.stdout:3022 read 0x8d2c9 thru 0x93d1a (0x6a52 bytes) 2024-09-16T13:29:16.920 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.920 INFO:teuthology.orchestra.run.smithi076.stdout:3025 write 0x4265139 thru 0x427254b (0xd413 bytes) 2024-09-16T13:29:16.926 INFO:teuthology.orchestra.run.smithi076.stdout:3027 punch from 0x709ef75 to 0x70a8250, (0x92db bytes) 2024-09-16T13:29:16.926 INFO:teuthology.orchestra.run.smithi076.stdout:3028 punch from 0x8521622 to 0x852b088, (0x9a66 bytes) 2024-09-16T13:29:16.926 INFO:teuthology.orchestra.run.smithi076.stdout:3029 write 0xdd5b5ba thru 0xdd61473 (0x5eba bytes) 2024-09-16T13:29:16.930 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.930 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.930 INFO:teuthology.orchestra.run.smithi076.stdout:3033 read 0x9cc6958 thru 0x9ccacc0 (0x4369 bytes) 2024-09-16T13:29:16.930 INFO:teuthology.orchestra.run.smithi076.stdout:3034 write 0xcdee096 thru 0xcdf2c09 (0x4b74 bytes) 2024-09-16T13:29:16.934 INFO:teuthology.orchestra.run.smithi076.stdout:3036 write 0xb771015 thru 0xb77d876 (0xc862 bytes) 2024-09-16T13:29:16.935 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:16.933+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:16.938 INFO:teuthology.orchestra.run.smithi076.stdout:3038 writesame 0x7f29dba thru 0x7f2a920 data_size 0x7(0xb67 bytes) 2024-09-16T13:29:16.946 INFO:teuthology.orchestra.run.smithi076.stdout:3040 write 0xea15390 thru 0xea1d47c (0x80ed bytes) 2024-09-16T13:29:16.951 INFO:teuthology.orchestra.run.smithi076.stdout:3041 writesame 0x74d8230 thru 0x74d9c94 data_size 0x1(0x1a65 bytes) 2024-09-16T13:29:16.963 INFO:teuthology.orchestra.run.smithi076.stdout:3042 trunc from 0xea1d47d to 0x90539de 2024-09-16T13:29:16.976 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:16.981 INFO:teuthology.orchestra.run.smithi076.stdout:3049 trunc from 0x90539de to 0xcd4a524 2024-09-16T13:29:16.986 INFO:teuthology.orchestra.run.smithi076.stdout:3050 write 0x20c58b thru 0x21428e (0x7d04 bytes) 2024-09-16T13:29:17.002 INFO:teuthology.orchestra.run.smithi076.stdout:3051 trunc from 0xcd4a524 to 0x5ea8839 2024-09-16T13:29:17.017 INFO:teuthology.orchestra.run.smithi076.stdout:3053 trunc from 0x5ea8839 to 0xb740a34 2024-09-16T13:29:17.022 INFO:teuthology.orchestra.run.smithi076.stdout:3055 write 0x8e174 thru 0x992cc (0xb159 bytes) 2024-09-16T13:29:17.028 INFO:teuthology.orchestra.run.smithi076.stdout:3058 read 0x8dd1ad9 thru 0x8ddd272 (0xb79a bytes) 2024-09-16T13:29:17.031 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.031 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.035 INFO:teuthology.orchestra.run.smithi076.stdout:3061 trunc from 0xb740a34 to 0x6235902 2024-09-16T13:29:17.036 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:17.033+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:17.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:17.051 INFO:teuthology.orchestra.run.smithi076.stdout:3062 write 0xb39156a thru 0xb3a07e4 (0xf27b bytes) 2024-09-16T13:29:17.055 INFO:teuthology.orchestra.run.smithi076.stdout:3064 punch from 0x2fbbb59 to 0x2fc6147, (0xa5ee bytes) 2024-09-16T13:29:17.055 INFO:teuthology.orchestra.run.smithi076.stdout:3065 read 0x3238177 thru 0x32461ae (0xe038 bytes) 2024-09-16T13:29:17.055 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.055 INFO:teuthology.orchestra.run.smithi076.stdout:3067 write 0x52f3dd1 thru 0x53037d0 (0xfa00 bytes) 2024-09-16T13:29:17.061 INFO:teuthology.orchestra.run.smithi076.stdout:3068 read 0x53da139 thru 0x53dbdb3 (0x1c7b bytes) 2024-09-16T13:29:17.061 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.061 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.061 INFO:teuthology.orchestra.run.smithi076.stdout:3071 writesame 0x516839 thru 0x52380a data_size 0x3(0xcfd2 bytes) 2024-09-16T13:29:17.070 INFO:teuthology.orchestra.run.smithi076.stdout:3072 punch from 0x6172722 to 0x617bb13, (0x93f1 bytes) 2024-09-16T13:29:17.071 INFO:teuthology.orchestra.run.smithi076.stdout:3073 read 0x47a9e34 thru 0x47b43e0 (0xa5ad bytes) 2024-09-16T13:29:17.071 INFO:teuthology.orchestra.run.smithi076.stdout:3075 read 0x82cff3c thru 0x82d728c (0x7351 bytes) 2024-09-16T13:29:17.071 INFO:teuthology.orchestra.run.smithi076.stdout:3076 read 0x1a8d44d thru 0x1a99ca0 (0xc854 bytes) 2024-09-16T13:29:17.072 INFO:teuthology.orchestra.run.smithi076.stdout:3077 write 0xc39bd2 thru 0xc49a21 (0xfe50 bytes) 2024-09-16T13:29:17.084 INFO:teuthology.orchestra.run.smithi076.stdout:3078 trunc from 0xb3a07e5 to 0xec07c60 2024-09-16T13:29:17.094 INFO:teuthology.orchestra.run.smithi076.stdout:3080 trunc from 0xec07c60 to 0xb79846e 2024-09-16T13:29:17.099 INFO:teuthology.orchestra.run.smithi076.stdout:3081 write 0x672589 thru 0x68116d (0xebe5 bytes) 2024-09-16T13:29:17.106 INFO:teuthology.orchestra.run.smithi076.stdout:3082 read 0xe9340e thru 0xe9d091 (0x9c84 bytes) 2024-09-16T13:29:17.106 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.106 INFO:teuthology.orchestra.run.smithi076.stdout:3084 write 0x789e8f thru 0x78e917 (0x4a89 bytes) 2024-09-16T13:29:17.109 INFO:teuthology.orchestra.run.smithi076.stdout:3086 writesame 0x1a3c369 thru 0x1a46799 data_size 0x1(0xa431 bytes) 2024-09-16T13:29:17.137 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:17.134+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:17.157 INFO:teuthology.orchestra.run.smithi076.stdout:3087 write 0x542f5f2 thru 0x54358e2 (0x62f1 bytes) 2024-09-16T13:29:17.160 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.160 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.162 INFO:teuthology.orchestra.run.smithi076.stdout:3088 read 0x9ee7988 thru 0x9ef2533 (0xabac bytes) 2024-09-16T13:29:17.162 INFO:teuthology.orchestra.run.smithi076.stdout:3090 writesame 0xabdd1d thru 0xabfdfb data_size 0x21(0x20df bytes) 2024-09-16T13:29:17.165 INFO:teuthology.orchestra.run.smithi076.stdout:3091 write 0x986ff0b thru 0x987b2ad (0xb3a3 bytes) 2024-09-16T13:29:17.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:17.174 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.174 INFO:teuthology.orchestra.run.smithi076.stdout:3093 write 0x6055c32 thru 0x6064623 (0xe9f2 bytes) 2024-09-16T13:29:17.181 INFO:teuthology.orchestra.run.smithi076.stdout:3094 writesame 0x157b57a thru 0x1585cdb data_size 0x19(0xa762 bytes) 2024-09-16T13:29:17.189 INFO:teuthology.orchestra.run.smithi076.stdout:3095 read 0x3292d99 thru 0x32971f5 (0x445d bytes) 2024-09-16T13:29:17.189 INFO:teuthology.orchestra.run.smithi076.stdout:3096 read 0x2baf65a thru 0x2bb3bc1 (0x4568 bytes) 2024-09-16T13:29:17.189 INFO:teuthology.orchestra.run.smithi076.stdout:3097 write 0x8976ff1 thru 0x898131b (0xa32b bytes) 2024-09-16T13:29:17.197 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.203 INFO:teuthology.orchestra.run.smithi076.stdout:3099 write 0xcb046e7 thru 0xcb06a2f (0x2349 bytes) 2024-09-16T13:29:17.206 INFO:teuthology.orchestra.run.smithi076.stdout:3101 read 0x5bcf401 thru 0x5bd2d06 (0x3906 bytes) 2024-09-16T13:29:17.213 INFO:teuthology.orchestra.run.smithi076.stdout:3104 writesame 0xe290084 thru 0xe29eccf data_size 0xc(0xec4c bytes) 2024-09-16T13:29:17.213 INFO:teuthology.orchestra.run.smithi076.stdout:3105 write 0x93cc4b0 thru 0x93d40de (0x7c2f bytes) 2024-09-16T13:29:17.217 INFO:teuthology.orchestra.run.smithi076.stdout:3106 trunc from 0xe29ecd0 to 0xdebe1ac 2024-09-16T13:29:17.236 INFO:teuthology.orchestra.run.smithi076.stdout:3107 trunc from 0xdebe1ac to 0x1846ba2 2024-09-16T13:29:17.237 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:29:17.238 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:17.235+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:17.265 INFO:teuthology.orchestra.run.smithi076.stdout:3109 punch from 0xd2867e to 0xd2efa2, (0x6924 bytes) 2024-09-16T13:29:17.271 INFO:teuthology.orchestra.run.smithi076.stdout:3110 writesame 0x215c559 thru 0x2163e94 data_size 0x2(0x793c bytes) 2024-09-16T13:29:17.271 INFO:teuthology.orchestra.run.smithi076.stdout:3111 read 0x17ff5d8 thru 0x18058e6 (0x630f bytes) 2024-09-16T13:29:17.272 INFO:teuthology.orchestra.run.smithi076.stdout:3112 read 0x1623506 thru 0x162d3fe (0x9ef9 bytes) 2024-09-16T13:29:17.274 INFO:teuthology.orchestra.run.smithi076.stdout:3113 trunc from 0x2163e95 to 0x406e3be 2024-09-16T13:29:17.289 INFO:teuthology.orchestra.run.smithi076.stdout:3115 write 0x8e93dba thru 0x8e952a0 (0x14e7 bytes) 2024-09-16T13:29:17.291 INFO:teuthology.orchestra.run.smithi076.stdout:3117 read 0x65bc77d thru 0x65bd2be (0xb42 bytes) 2024-09-16T13:29:17.295 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.295 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.299 INFO:teuthology.orchestra.run.smithi076.stdout:3118 write 0xb457542 thru 0xb466fe4 (0xfaa3 bytes) 2024-09-16T13:29:17.304 INFO:teuthology.orchestra.run.smithi076.stdout:3121 punch from 0x53e60bb to 0x53e6be0, (0xb25 bytes) 2024-09-16T13:29:17.304 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.304 INFO:teuthology.orchestra.run.smithi076.stdout:3123 write 0x714ea94 thru 0x71513d0 (0x293d bytes) 2024-09-16T13:29:17.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:17.307 INFO:teuthology.orchestra.run.smithi076.stdout:3124 punch from 0x3349fde to 0x334c726, (0x2748 bytes) 2024-09-16T13:29:17.307 INFO:teuthology.orchestra.run.smithi076.stdout:3126 read 0x9e14cae thru 0x9e2311e (0xe471 bytes) 2024-09-16T13:29:17.308 INFO:teuthology.orchestra.run.smithi076.stdout:3128 read 0x9c747fe thru 0x9c80f8e (0xc791 bytes) 2024-09-16T13:29:17.308 INFO:teuthology.orchestra.run.smithi076.stdout:3129 punch from 0x37b1844 to 0x37bb225, (0x99e1 bytes) 2024-09-16T13:29:17.308 INFO:teuthology.orchestra.run.smithi076.stdout:3130 write 0x6ee70f2 thru 0x6ef57e9 (0xe6f8 bytes) 2024-09-16T13:29:17.313 INFO:teuthology.orchestra.run.smithi076.stdout:3131 writesame 0x918e5c6 thru 0x9199ee2 data_size 0x1(0xb91d bytes) 2024-09-16T13:29:17.338 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:17.335+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:17.363 INFO:teuthology.orchestra.run.smithi076.stdout:3132 punch from 0x51b270b to 0x51bc9a6, (0xa29b bytes) 2024-09-16T13:29:17.370 INFO:teuthology.orchestra.run.smithi076.stdout:3133 trunc from 0xb466fe5 to 0x55d41ac 2024-09-16T13:29:17.381 INFO:teuthology.orchestra.run.smithi076.stdout:3134 write 0x166c92d thru 0x1674fd1 (0x86a5 bytes) 2024-09-16T13:29:17.387 INFO:teuthology.orchestra.run.smithi076.stdout:3135 punch from 0x377e3fb to 0x3789ea3, (0xbaa8 bytes) 2024-09-16T13:29:17.388 INFO:teuthology.orchestra.run.smithi076.stdout:3140 read 0x4fefa72 thru 0x4fff6f6 (0xfc85 bytes) 2024-09-16T13:29:17.388 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.388 INFO:teuthology.orchestra.run.smithi076.stdout:3142 read 0xcdfe9f thru 0xce784d (0x79af bytes) 2024-09-16T13:29:17.388 INFO:teuthology.orchestra.run.smithi076.stdout:3144 write 0xd49e7 thru 0xe3b01 (0xf11b bytes) 2024-09-16T13:29:17.393 INFO:teuthology.orchestra.run.smithi076.stdout:3146 read 0x3b0111 thru 0x3b3d6b (0x3c5b bytes) 2024-09-16T13:29:17.395 INFO:teuthology.orchestra.run.smithi076.stdout:3147 trunc from 0x55d41ac to 0x770c79d 2024-09-16T13:29:17.411 INFO:teuthology.orchestra.run.smithi076.stdout:3148 write 0xbc32a47 thru 0xbc415ca (0xeb84 bytes) 2024-09-16T13:29:17.417 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.417 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:17.430 INFO:teuthology.orchestra.run.smithi076.stdout:3149 trunc from 0xbc415cb to 0x29a16d5 2024-09-16T13:29:17.439 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:17.436+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:17.440 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.440 INFO:teuthology.orchestra.run.smithi076.stdout:3151 read 0x2311934 thru 0x231c06d (0xa73a bytes) 2024-09-16T13:29:17.457 INFO:teuthology.orchestra.run.smithi076.stdout:3152 write 0xc4aa6f3 thru 0xc4b5696 (0xafa4 bytes) 2024-09-16T13:29:17.460 INFO:teuthology.orchestra.run.smithi076.stdout:3153 punch from 0xb732c81 to 0xb73bc29, (0x8fa8 bytes) 2024-09-16T13:29:17.460 INFO:teuthology.orchestra.run.smithi076.stdout:3154 write 0x4f4f748 thru 0x4f56f8e (0x7847 bytes) 2024-09-16T13:29:17.465 INFO:teuthology.orchestra.run.smithi076.stdout:3156 read 0x8f8bb42 thru 0x8f9003a (0x44f9 bytes) 2024-09-16T13:29:17.465 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.465 INFO:teuthology.orchestra.run.smithi076.stdout:3159 write 0xa46f423 thru 0xa47bcb6 (0xc894 bytes) 2024-09-16T13:29:17.470 INFO:teuthology.orchestra.run.smithi076.stdout:3160 write 0x281dc8 thru 0x28cbf0 (0xae29 bytes) 2024-09-16T13:29:17.474 INFO:teuthology.orchestra.run.smithi076.stdout:3161 write 0x7670c4e thru 0x7680c2b (0xffde bytes) 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:3163 punch from 0xa96d5ba to 0xa971c9c, (0x46e2 bytes) 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:3164 read 0xb051e50 thru 0xb055b97 (0x3d48 bytes) 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:3166 read 0x19564a3 thru 0x1956f4d (0xaab bytes) 2024-09-16T13:29:17.480 INFO:teuthology.orchestra.run.smithi076.stdout:3167 clone 25 order 20 su 8192 sc 9 2024-09-16T13:29:17.529 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.529 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.538 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:17.539 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:17.537+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:17.636 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.636 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.640 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:17.637+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:17.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:17.740 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:17.738+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:17.782 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.782 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:17.842 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:17.839+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:17.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:17.930 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:17.930 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:17.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:17.983 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:17.980+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:18.037 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:18.037 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:18.043 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:18.040+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:18.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:18.046 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:18.046 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:18.062 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:18.062 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:18.077 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:18.091 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:18.107 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:18.107 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:18.123 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:18.138 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:18.138 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:18.144 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:18.142+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:18.154 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:18.249 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:18.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:18.249 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-16T13:29:18.249 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:29:18.249 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:29:18.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-16T13:29:18.251 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:18.251 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 43.544913063999999, 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:18.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:18.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:18.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 43.454382658999997, 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:18.259 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:18.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.265 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 43.427253489000002, 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:18.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:18.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.011491337000002, 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:18.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:18.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.941155055000003, 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:18.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:18.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.498033309, 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:18.279 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:18.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:18.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.487944041999999, 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.341155407999999, 2024-09-16T13:29:18.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:18.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50162 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 581632~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935844667, 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019455525000000001, 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.296 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50162 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248555+0000", 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248557+0000", 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7850000000000001e-06 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248562+0000", 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8579999999999996e-06 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248566+0000", 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.207e-06 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267854+0000", 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019288254000000001 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267938+0000", 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3678999999999996e-05 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268010+0000", 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2416000000000006e-05 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50163 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 598016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935829816, 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019564818000000001, 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50163 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248570+0000", 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:29:18.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248577+0000", 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4320000000000002e-06 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248584+0000", 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248587+0000", 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3790000000000002e-06 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268040+0000", 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019453427999999998 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268096+0000", 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5404e-05 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268134+0000", 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50164 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 614400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:18.306 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935808141000001, 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019695615, 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50164 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248593+0000", 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9300000000000002e-06 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248599+0000", 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7950000000000004e-06 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248603+0000", 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268171+0000", 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019568232000000001 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268243+0000", 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1872000000000006e-05 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268287+0000", 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3300000000000002e-05 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.311 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50165 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 630784~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935791953000001, 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019831332, 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.312 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50165 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248608+0000", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248610+0000", 2024-09-16T13:29:18.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.807e-06 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248619+0000", 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4779999999999997e-06 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248623+0000", 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0489999999999997e-06 2024-09-16T13:29:18.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268313+0000", 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019690151999999999 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268379+0000", 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5792999999999998e-05 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268439+0000", 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537000000000003e-05 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50166 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:18.317 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935772182000001, 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019843890999999999, 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50166 2024-09-16T13:29:18.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.65e-07 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248629+0000", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.854e-06 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248635+0000", 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248645+0000", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7599999999999997e-06 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268466+0000", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019821442000000002 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268471+0000", 2024-09-16T13:29:18.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.775e-06 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50167 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935750552, 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019857698999999999, 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.322 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50167 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248651+0000", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.181e-06 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248657+0000", 2024-09-16T13:29:18.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0939999999999996e-06 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248661+0000", 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268499+0000", 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019837751000000001 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268506+0000", 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7440000000000007e-06 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50168 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.935734839, 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019875693999999999, 2024-09-16T13:29:18.326 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50168 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248665+0000", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248667+0000", 2024-09-16T13:29:18.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.866e-06 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248672+0000", 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3410000000000002e-06 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248675+0000", 2024-09-16T13:29:18.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.264e-06 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268536+0000", 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019860932000000001 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268540+0000", 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50929 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 327680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.887722854, 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019357040999999998, 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50929 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296677+0000", 2024-09-16T13:29:18.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9900000000000002e-07 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296678+0000", 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.401e-06 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296683+0000", 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296686+0000", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8949999999999998e-06 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315967+0000", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019280701000000001 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316000+0000", 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3549e-05 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316033+0000", 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3272999999999998e-05 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50930 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 344064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.887710511, 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019423292000000002, 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.335 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50930 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5499999999999999e-07 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296690+0000", 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.575e-06 2024-09-16T13:29:18.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296696+0000", 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3460000000000001e-06 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296704+0000", 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6319999999999997e-06 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316046+0000", 2024-09-16T13:29:18.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019341649999999998 2024-09-16T13:29:18.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316079+0000", 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2898999999999998e-05 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316112+0000", 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2935000000000002e-05 2024-09-16T13:29:18.340 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.2092310150000003, 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.341 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:18.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:18.347 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2610720419999999, 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:29:18.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:29:18.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:58637 4.15 e74/70 ondisk, result = 0)", 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2516002130000001, 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026177753000000002, 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.354 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932800+0000", 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932807+0000", 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1659999999999999e-06 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932826+0000", 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9256999999999999e-05 2024-09-16T13:29:18.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932835+0000", 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6239999999999997e-06 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958962+0000", 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026126531000000001 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958971+0000", 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5839999999999995e-06 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958977+0000", 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1600000000000003e-06 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.358 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:18.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:18.360 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:18.288+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:18.361 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone24 from 0xc4b5697 (overlap 0x1846ba2) to 0x10fe4f1 2024-09-16T13:29:18.361 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:18.357+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:18.461 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:18.461 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:18.461 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:18.461 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:18.462 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:18.459+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:18.466 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:18.467 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:29:18.467 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:18.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:18.477 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-09-16T13:29:18.481 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.481 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.481 INFO:teuthology.orchestra.run.smithi076.stdout:3170 write 0x578dc82 thru 0x579ba40 (0xddbf bytes) 2024-09-16T13:29:18.496 INFO:teuthology.orchestra.run.smithi076.stdout:3171 write 0xbc8ea2f thru 0xbc94ebf (0x6491 bytes) 2024-09-16T13:29:18.504 INFO:teuthology.orchestra.run.smithi076.stdout:3172 punch from 0x781b80 to 0x78582f, (0x3caf bytes) 2024-09-16T13:29:18.504 INFO:teuthology.orchestra.run.smithi076.stdout:3173 write 0x994fd17 thru 0x9956fb8 (0x72a2 bytes) 2024-09-16T13:29:18.515 INFO:teuthology.orchestra.run.smithi076.stdout:3174 write 0x62b856c thru 0x62c518c (0xcc21 bytes) 2024-09-16T13:29:18.540 INFO:teuthology.orchestra.run.smithi076.stdout:3175 trunc from 0xc4b5697 to 0x2a35d54 2024-09-16T13:29:18.562 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:18.559+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:18.578 INFO:teuthology.orchestra.run.smithi076.stdout:3176 trunc from 0x2a35d54 to 0xd8e05d3 2024-09-16T13:29:18.583 INFO:teuthology.orchestra.run.smithi076.stdout:3177 writesame 0x42a7999 thru 0x42b1a55 data_size 0x1(0xa0bd bytes) 2024-09-16T13:29:18.609 INFO:teuthology.orchestra.run.smithi076.stdout:3178 read 0x2df8075 thru 0x2dfb07b (0x3007 bytes) 2024-09-16T13:29:18.616 INFO:teuthology.orchestra.run.smithi076.stdout:3179 write 0xe4c25e0 thru 0xe4d15cb (0xefec bytes) 2024-09-16T13:29:18.625 INFO:teuthology.orchestra.run.smithi076.stdout:3180 write 0x54afaf5 thru 0x54b610d (0x6619 bytes) 2024-09-16T13:29:18.630 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:18.630 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:18.630 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:18.630 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:18.631 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:18.634 INFO:teuthology.orchestra.run.smithi076.stdout:3181 writesame 0xd1bb05 thru 0xd1f3fa data_size 0x17(0x38f6 bytes) 2024-09-16T13:29:18.640 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:18.656 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.656 INFO:teuthology.orchestra.run.smithi076.stdout:3184 read 0xc59c5c9 thru 0xc5aa812 (0xe24a bytes) 2024-09-16T13:29:18.656 INFO:teuthology.orchestra.run.smithi076.stdout:3185 writesame 0x12083ae thru 0x120cb36 data_size 0x1(0x4789 bytes) 2024-09-16T13:29:18.679 INFO:teuthology.orchestra.run.smithi076.stdout:3186 write 0x1593ea thru 0x15ce16 (0x3a2d bytes) 2024-09-16T13:29:18.682 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:18.679+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:18.701 INFO:teuthology.orchestra.run.smithi076.stdout:3187 write 0xb16236d thru 0xb1675b4 (0x5248 bytes) 2024-09-16T13:29:18.708 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.708 INFO:teuthology.orchestra.run.smithi076.stdout:3190 writesame 0xdbdec09 thru 0xdbed465 data_size 0x1(0xe85d bytes) 2024-09-16T13:29:18.735 INFO:teuthology.orchestra.run.smithi076.stdout:3191 writesame 0x78cba5d thru 0x78d9d60 data_size 0xa7(0xe304 bytes) 2024-09-16T13:29:18.747 INFO:teuthology.orchestra.run.smithi076.stdout:3192 write 0x57d1bfd thru 0x57d6efd (0x5301 bytes) 2024-09-16T13:29:18.753 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.753 INFO:teuthology.orchestra.run.smithi076.stdout:3195 read 0x4e00eca thru 0x4e0cc27 (0xbd5e bytes) 2024-09-16T13:29:18.753 INFO:teuthology.orchestra.run.smithi076.stdout:3196 write 0x7833e53 thru 0x78412b8 (0xd466 bytes) 2024-09-16T13:29:18.759 INFO:teuthology.orchestra.run.smithi076.stdout:3197 read 0xbe370a5 thru 0xbe3efcd (0x7f29 bytes) 2024-09-16T13:29:18.759 INFO:teuthology.orchestra.run.smithi076.stdout:3198 write 0x2b54af1 thru 0x2b56f92 (0x24a2 bytes) 2024-09-16T13:29:18.762 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.762 INFO:teuthology.orchestra.run.smithi076.stdout:3200 write 0x358e9 thru 0x3b7f3 (0x5f0b bytes) 2024-09-16T13:29:18.765 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:18.762+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:18.778 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.778 INFO:teuthology.orchestra.run.smithi076.stdout:3202 write 0x6c20cfd thru 0x6c2ed5a (0xe05e bytes) 2024-09-16T13:29:18.788 INFO:teuthology.orchestra.run.smithi076.stdout:3203 punch from 0xd1e6cc4 to 0xd1f48e8, (0xdc24 bytes) 2024-09-16T13:29:18.788 INFO:teuthology.orchestra.run.smithi076.stdout:3204 write 0xd538245 thru 0xd53bdb4 (0x3b70 bytes) 2024-09-16T13:29:18.791 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.792 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.792 INFO:teuthology.orchestra.run.smithi076.stdout:3207 write 0x46160fe thru 0x461bedd (0x5de0 bytes) 2024-09-16T13:29:18.794 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.794 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.799 INFO:teuthology.orchestra.run.smithi076.stdout:3210 trunc from 0xe4d15cc to 0xa96f5c8 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.116111879999998, 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:18.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.106540402, 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:18.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7495 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 11862016~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.259143989999998, 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036999569000000003, 2024-09-16T13:29:18.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7495 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543987+0000", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0100000000000001e-07 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543988+0000", 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5199999999999995e-07 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543992+0000", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5609999999999999e-06 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543997+0000", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.66e-06 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580618+0000", 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036620898999999998 2024-09-16T13:29:18.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580677+0000", 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9122000000000001e-05 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.580987+0000", 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030927400000000001 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.259131719999999, 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.826 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.259120875000001, 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:29:18.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.259111461000003, 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8155 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6291456~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512872328, 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036820862000000003, 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8155 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290259+0000", 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290260+0000", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7100000000000001e-07 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290263+0000", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.495e-06 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290265+0000", 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:29:18.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326654+0000", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036389183999999998 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.326741+0000", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6904999999999995e-05 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327079+0000", 2024-09-16T13:29:18.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033837599999999999 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512863983000003, 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:18.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512853288999999, 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:29:18.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512844301000001, 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.853 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512836559999997, 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512828001000003, 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.858 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512819124000004, 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:18.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:18.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512805258999997, 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:18.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:18.871 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512796792000003, 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:18.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.512787989000003, 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:18.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:18.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.889317931000001, 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.880 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:18.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:18.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.87608925, 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:18.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.362263061, 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:29:18.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:29:18.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.345207182999999, 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:29:18.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:18.899 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:18.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:18.902 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:18.898+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:18.902 INFO:teuthology.orchestra.run.smithi076.stdout:3211 read 0xa90f6d6 thru 0xa915605 (0x5f30 bytes) 2024-09-16T13:29:18.902 INFO:teuthology.orchestra.run.smithi076.stdout:3213 read 0x6e5ee74 thru 0x6e6693a (0x7ac7 bytes) 2024-09-16T13:29:18.902 INFO:teuthology.orchestra.run.smithi076.stdout:3214 trunc from 0xa96f5c8 to 0x6a2d968 2024-09-16T13:29:18.902 INFO:teuthology.orchestra.run.smithi076.stdout:3215 trunc from 0x6a2d968 to 0xcb88785 2024-09-16T13:29:18.903 INFO:teuthology.orchestra.run.smithi076.stdout:3216 punch from 0xb9f9596 to 0xba063bb, (0xce25 bytes) 2024-09-16T13:29:18.903 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.903 INFO:teuthology.orchestra.run.smithi076.stdout:3220 write 0xc56107b thru 0xc56d3a5 (0xc32b bytes) 2024-09-16T13:29:18.903 INFO:teuthology.orchestra.run.smithi076.stdout:3222 trunc from 0xcb88785 to 0xe2eff8 2024-09-16T13:29:18.904 INFO:teuthology.orchestra.run.smithi076.stdout:3223 read 0x2024da thru 0x20acbd (0x87e4 bytes) 2024-09-16T13:29:18.905 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.905 INFO:teuthology.orchestra.run.smithi076.stdout:3226 read 0xbfc176 thru 0xc09562 (0xd3ed bytes) 2024-09-16T13:29:18.905 INFO:teuthology.orchestra.run.smithi076.stdout:3227 write 0x8369a0 thru 0x845bcb (0xf22c bytes) 2024-09-16T13:29:18.939 INFO:teuthology.orchestra.run.smithi076.stdout:3229 writesame 0x706c255 thru 0x707b670 data_size 0x22(0xf41c bytes) 2024-09-16T13:29:18.940 INFO:teuthology.orchestra.run.smithi076.stdout:3232 read 0x14fe382 thru 0x150c011 (0xdc90 bytes) 2024-09-16T13:29:18.940 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.947 INFO:teuthology.orchestra.run.smithi076.stdout:3234 write 0xa043514 thru 0xa049bb6 (0x66a3 bytes) 2024-09-16T13:29:18.956 INFO:teuthology.orchestra.run.smithi076.stdout:3237 read 0x5367d56 thru 0x5368f17 (0x11c2 bytes) 2024-09-16T13:29:18.956 INFO:teuthology.orchestra.run.smithi076.stdout:3238 write 0x2d751a5 thru 0x2d77938 (0x2794 bytes) 2024-09-16T13:29:18.961 INFO:teuthology.orchestra.run.smithi076.stdout:3239 trunc from 0xa049bb7 to 0x84d9306 2024-09-16T13:29:18.971 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.971 INFO:teuthology.orchestra.run.smithi076.stdout:3243 read 0x67a484e thru 0x67b1606 (0xcdb9 bytes) 2024-09-16T13:29:18.971 INFO:teuthology.orchestra.run.smithi076.stdout:3244 write 0x459a4bd thru 0x45a5664 (0xb1a8 bytes) 2024-09-16T13:29:18.978 INFO:teuthology.orchestra.run.smithi076.stdout:3246 write 0x50d7bce thru 0x50dae92 (0x32c5 bytes) 2024-09-16T13:29:18.982 INFO:teuthology.orchestra.run.smithi076.stdout:3251 write 0x70fd98b thru 0x71004fd (0x2b73 bytes) 2024-09-16T13:29:18.986 INFO:teuthology.orchestra.run.smithi076.stdout:3252 read 0x196a1cb thru 0x196e2a6 (0x40dc bytes) 2024-09-16T13:29:18.987 INFO:teuthology.orchestra.run.smithi076.stdout:3254 trunc from 0x84d9306 to 0x8e94bc4 2024-09-16T13:29:18.993 INFO:teuthology.orchestra.run.smithi076.stdout:3255 trunc from 0x8e94bc4 to 0x9b5b25b 2024-09-16T13:29:18.999 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.999 INFO:teuthology.orchestra.run.smithi076.stdout:3257 read 0x53bea75 thru 0x53ccbd7 (0xe163 bytes) 2024-09-16T13:29:18.999 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:18.999 INFO:teuthology.orchestra.run.smithi076.stdout:3261 trunc from 0x9b5b25b to 0xa0d16d8 2024-09-16T13:29:19.001 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:18.999+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:19.003 INFO:teuthology.orchestra.run.smithi076.stdout:3262 writesame 0x58f6e5a thru 0x59063c2 data_size 0x23(0xf569 bytes) 2024-09-16T13:29:19.017 INFO:teuthology.orchestra.run.smithi076.stdout:3263 read 0xd3b3f1 thru 0xd490a4 (0xdcb4 bytes) 2024-09-16T13:29:19.017 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.023 INFO:teuthology.orchestra.run.smithi076.stdout:3265 trunc from 0xa0d16d8 to 0x5cfbd16 2024-09-16T13:29:19.032 INFO:teuthology.orchestra.run.smithi076.stdout:3266 read 0x271af15 thru 0x2727c89 (0xcd75 bytes) 2024-09-16T13:29:19.032 INFO:teuthology.orchestra.run.smithi076.stdout:3267 read 0x4a2e39a thru 0x4a30107 (0x1d6e bytes) 2024-09-16T13:29:19.033 INFO:teuthology.orchestra.run.smithi076.stdout:3269 read 0x352a98d thru 0x353182b (0x6e9f bytes) 2024-09-16T13:29:19.033 INFO:teuthology.orchestra.run.smithi076.stdout:3270 write 0x2645ffe thru 0x2651c15 (0xbc18 bytes) 2024-09-16T13:29:19.045 INFO:teuthology.orchestra.run.smithi076.stdout:3273 read 0x3903039 thru 0x39063a9 (0x3371 bytes) 2024-09-16T13:29:19.045 INFO:teuthology.orchestra.run.smithi076.stdout:3274 punch from 0x1da4338 to 0x1dab3a8, (0x7070 bytes) 2024-09-16T13:29:19.045 INFO:teuthology.orchestra.run.smithi076.stdout:3275 read 0x324dac3 thru 0x325d5b0 (0xfaee bytes) 2024-09-16T13:29:19.050 INFO:teuthology.orchestra.run.smithi076.stdout:3276 write 0x63f5277 thru 0x63f7d8f (0x2b19 bytes) 2024-09-16T13:29:19.055 INFO:teuthology.orchestra.run.smithi076.stdout:3277 trunc from 0x63f7d90 to 0x5259c4a 2024-09-16T13:29:19.056 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:19.056 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:19.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:62225 4.8 e77/70 4:12c321bf:::rbd_object_map.11a911a8a02a:head v 77'468, mlcod=77'466)", 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:19.055023+0000", 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00035161099999999999, 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037468000000000001, 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055023+0000", 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055023+0000", 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055024+0000", 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4199999999999995e-07 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055028+0000", 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055038+0000", 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6840000000000008e-06 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055043+0000", 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0340000000000004e-06 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055075+0000", 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2441000000000003e-05 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:19.055089+0000", 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3570000000000001e-05 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:29:19.060 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:19.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:19.068 INFO:teuthology.orchestra.run.smithi076.stdout:3278 punch from 0x3335fdd to 0x3343b3b, (0xdb5e bytes) 2024-09-16T13:29:19.068 INFO:teuthology.orchestra.run.smithi076.stdout:3279 punch from 0x6b2f8 to 0x6d074, (0x1d7c bytes) 2024-09-16T13:29:19.068 INFO:teuthology.orchestra.run.smithi076.stdout:3280 punch from 0x4e57d1e to 0x4e5a554, (0x2836 bytes) 2024-09-16T13:29:19.084 INFO:teuthology.orchestra.run.smithi076.stdout:3281 write 0xe70b0f9 thru 0xe711e14 (0x6d1c bytes) 2024-09-16T13:29:19.093 INFO:teuthology.orchestra.run.smithi076.stdout:3282 write 0xb3c15e3 thru 0xb3c5e25 (0x4843 bytes) 2024-09-16T13:29:19.100 INFO:teuthology.orchestra.run.smithi076.stdout:3283 write 0x32e9cc0 thru 0x32f34df (0x9820 bytes) 2024-09-16T13:29:19.103 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:19.100+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:19.122 INFO:teuthology.orchestra.run.smithi076.stdout:3284 trunc from 0xe711e15 to 0x4160041 2024-09-16T13:29:19.137 INFO:teuthology.orchestra.run.smithi076.stdout:3285 trunc from 0x4160041 to 0x44266ad 2024-09-16T13:29:19.142 INFO:teuthology.orchestra.run.smithi076.stdout:3286 writesame 0x932f74 thru 0x942904 data_size 0x1(0xf991 bytes) 2024-09-16T13:29:19.161 INFO:teuthology.orchestra.run.smithi076.stdout:3287 writesame 0x3b334ea thru 0x3b3baa4 data_size 0x29(0x85bb bytes) 2024-09-16T13:29:19.169 INFO:teuthology.orchestra.run.smithi076.stdout:3288 write 0x21f284c thru 0x21ff9ad (0xd162 bytes) 2024-09-16T13:29:19.179 INFO:teuthology.orchestra.run.smithi076.stdout:3289 trunc from 0x44266ad to 0x3916654 2024-09-16T13:29:19.180 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:19.180 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:19.180 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:19.180 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:19.180 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:19.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:19.203 INFO:teuthology.orchestra.run.smithi076.stdout:3292 write 0x8716ca1 thru 0x8722f60 (0xc2c0 bytes) 2024-09-16T13:29:19.211 INFO:teuthology.orchestra.run.smithi076.stdout:3293 write 0x682ff55 thru 0x68342d8 (0x4384 bytes) 2024-09-16T13:29:19.222 INFO:teuthology.orchestra.run.smithi076.stdout:3294 trunc from 0x8722f61 to 0x63f7f01 2024-09-16T13:29:19.232 INFO:teuthology.orchestra.run.smithi076.stdout:3295 read 0x1702aff thru 0x1708c67 (0x6169 bytes) 2024-09-16T13:29:19.236 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:19.232+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:19.237 INFO:teuthology.orchestra.run.smithi076.stdout:3297 trunc from 0x63f7f01 to 0x28d69fd 2024-09-16T13:29:19.254 INFO:teuthology.orchestra.run.smithi076.stdout:3298 read 0x8e7982 thru 0x8e7a65 (0xe4 bytes) 2024-09-16T13:29:19.255 INFO:teuthology.orchestra.run.smithi076.stdout:3299 read 0x1a5894e thru 0x1a5e29a (0x594d bytes) 2024-09-16T13:29:19.255 INFO:teuthology.orchestra.run.smithi076.stdout:3300 write 0xc3e29c thru 0xc47f63 (0x9cc8 bytes) 2024-09-16T13:29:19.257 INFO:teuthology.orchestra.run.smithi076.stdout:3301 punch from 0xb95ecd to 0xb9ecce, (0x8e01 bytes) 2024-09-16T13:29:19.257 INFO:teuthology.orchestra.run.smithi076.stdout:3302 write 0x264bcfb thru 0x26536f1 (0x79f7 bytes) 2024-09-16T13:29:19.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:29:19.268 INFO:teuthology.orchestra.run.smithi076.stdout:3303 trunc from 0x28d69fd to 0x9845ffb 2024-09-16T13:29:19.274 INFO:teuthology.orchestra.run.smithi076.stdout:3304 write 0x7a6fd73 thru 0x7a720c5 (0x2353 bytes) 2024-09-16T13:29:19.283 INFO:teuthology.orchestra.run.smithi076.stdout:3305 read 0x2d7affd thru 0x2d7c7c7 (0x17cb bytes) 2024-09-16T13:29:19.287 INFO:teuthology.orchestra.run.smithi076.stdout:3306 trunc from 0x9845ffb to 0x614a029 2024-09-16T13:29:19.298 INFO:teuthology.orchestra.run.smithi076.stdout:3308 read 0x30723b2 thru 0x307f47a (0xd0c9 bytes) 2024-09-16T13:29:19.304 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:19.304 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:19.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:19.304 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 74.010813189999993, 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:19.305 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:19.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:19.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:19.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:19.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:19.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:19.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:19.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:19.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:19.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:19.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.309462992999997, 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:29:19.320 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:29:19.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:29:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.309340597000002, 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.326 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:19.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.309165394999994, 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:19.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:19.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.338 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.309002781999993, 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:19.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:19.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:19.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308981735000003, 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:19.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:19.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.352 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308894254999998, 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:19.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308871745999994, 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.359 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:19.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308825419000001, 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:19.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:19.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:19.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308658661999999, 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.372 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:19.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:19.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.308643896999996, 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:19.378 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:19.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:19.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:19.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:19.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:19.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.387 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.963321579999999, 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:19.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:19.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.394 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.963283902000001, 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.395 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:19.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:19.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.401 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.963261089, 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.402 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:19.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:19.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.963175875999994, 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.409 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.415 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.656596950999997, 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:19.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.630688716999998, 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.420 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:19.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.561923561999997, 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:19.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.060380745, 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:19.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:19.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:61252 4.15 e77/70 4:ade293bf:::rbd_data.11a911a8a02a.0000000000000003:head v 77'331, mlcod=77'329)", 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:19.433 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.60753768100000005, 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076711440000000004, 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:19.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692478+0000", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.502e-06 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692676+0000", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019869500000000001 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692697+0000", 2024-09-16T13:29:19.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0480999999999999e-05 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692705+0000", 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4209999999999995e-06 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692762+0000", 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6536000000000002e-05 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:19.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692784+0000", 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2266999999999999e-05 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.700105+0000", 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073212650000000004 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.700147+0000", 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1977000000000001e-05 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:19.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:19.438 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:19.440 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:19.436+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:19.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:19.442 INFO:teuthology.orchestra.run.smithi076.stdout:3309 writesame 0x763ce3d thru 0x763e84c data_size 0x18(0x1a10 bytes) 2024-09-16T13:29:19.442 INFO:teuthology.orchestra.run.smithi076.stdout:3311 write 0x779f7ac thru 0x77a0e87 (0x16dc bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3312 write 0x4628231 thru 0x462c7a6 (0x4576 bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3314 trunc from 0x77a0e88 to 0x493ae07 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3315 write 0xdd5e1b1 thru 0xdd5e3a5 (0x1f5 bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3316 read 0xd3cd1a8 thru 0xd3d15b6 (0x440f bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3317 write 0x32c444f thru 0x32ccb8a (0x873c bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3318 read 0x6df9d62 thru 0x6e071f0 (0xd48f bytes) 2024-09-16T13:29:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:3319 punch from 0x625713e to 0x62602db, (0x919d bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3320 write 0xc470e00 thru 0xc480b2d (0xfd2e bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3322 trunc from 0xdd5e3a6 to 0xc7a61a9 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3323 trunc from 0xc7a61a9 to 0x5badf3f 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3324 read 0x37e4479 thru 0x37f3c2d (0xf7b5 bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3325 writesame 0x9c55cc7 thru 0x9c610aa data_size 0x1d(0xb3e4 bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3326 writesame 0x4373d85 thru 0x4379e35 data_size 0x25(0x60b1 bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3327 read 0x48be9b6 thru 0x48c63e7 (0x7a32 bytes) 2024-09-16T13:29:19.444 INFO:teuthology.orchestra.run.smithi076.stdout:3328 write 0xe108cb8 thru 0xe10f04b (0x6394 bytes) 2024-09-16T13:29:19.445 INFO:teuthology.orchestra.run.smithi076.stdout:3329 write 0x2caa6c4 thru 0x2cb438c (0x9cc9 bytes) 2024-09-16T13:29:19.445 INFO:teuthology.orchestra.run.smithi076.stdout:3330 read 0xae0d9db thru 0xae13066 (0x568c bytes) 2024-09-16T13:29:19.445 INFO:teuthology.orchestra.run.smithi076.stdout:3332 write 0x977172d thru 0x977bc52 (0xa526 bytes) 2024-09-16T13:29:19.450 INFO:teuthology.orchestra.run.smithi076.stdout:3333 read 0xb1812d6 thru 0xb184554 (0x327f bytes) 2024-09-16T13:29:19.450 INFO:teuthology.orchestra.run.smithi076.stdout:3334 read 0x20ab17e thru 0x20ab281 (0x104 bytes) 2024-09-16T13:29:19.450 INFO:teuthology.orchestra.run.smithi076.stdout:3335 trunc from 0xe10f04c to 0xdd1a3a9 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3336 read 0xd0850c9 thru 0xd08f7ad (0xa6e5 bytes) 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3337 punch from 0x153f01f to 0x1542bbc, (0x3b9d bytes) 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3342 punch from 0x8fcaa19 to 0x8fcd3ad, (0x2994 bytes) 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3344 read 0x5e778c0 thru 0x5e83c75 (0xc3b6 bytes) 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3346 read 0xbfc35f0 thru 0xbfc5c07 (0x2618 bytes) 2024-09-16T13:29:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:3347 write 0x68d7542 thru 0x68dc398 (0x4e57 bytes) 2024-09-16T13:29:19.469 INFO:teuthology.orchestra.run.smithi076.stdout:3348 read 0xccb6549 thru 0xccc0b4c (0xa604 bytes) 2024-09-16T13:29:19.469 INFO:teuthology.orchestra.run.smithi076.stdout:3349 write 0x33dbcc0 thru 0x33eb325 (0xf666 bytes) 2024-09-16T13:29:19.476 INFO:teuthology.orchestra.run.smithi076.stdout:3350 punch from 0x8a7371e to 0x8a7f316, (0xbbf8 bytes) 2024-09-16T13:29:19.476 INFO:teuthology.orchestra.run.smithi076.stdout:3351 read 0xc8e63d9 thru 0xc8f5955 (0xf57d bytes) 2024-09-16T13:29:19.476 INFO:teuthology.orchestra.run.smithi076.stdout:3352 clone 26 order 18 su 16384 sc 14 2024-09-16T13:29:19.540 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:19.537+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:19.547 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:19.547 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:19.547 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:19.547 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:19.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:19.641 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:19.638+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:19.694 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:19.694 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:19.694 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:19.694 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:19.694 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:19.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:19.747 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:19.745+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:19.801 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.632996249000001, 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.802 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:19.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.627520683999997, 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.808 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:19.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.867193876999998, 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.814 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:19.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:19.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.850655752000002, 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.508025869999997, 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:29:19.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.508008891999999, 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.828 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.832 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507992301999998, 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:19.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:19.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:19.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507969957, 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:19.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:19.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:19.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507951976999998, 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:19.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:19.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:19.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507936172999997, 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.850 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:19.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507920611999999, 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:19.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:19.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507902897999998, 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:19.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:19.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.507876314999997, 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.864 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:19.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:19.869 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.666322401999999, 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.870 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:19.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:19.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:19.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:19.878 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.468882831000002, 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:19.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.458384084000002, 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.885 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:29:19.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.894006224999998, 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:19.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.546716267000001, 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:29:19.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.546436371, 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:29:19.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:29:19.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:29:19.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:29:19.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:29:19.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:19.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 6.114744655, 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:19.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:19.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:29:19.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:29:19.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:19.916 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:19.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:19.918 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:19.915+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:20.019 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:20.016+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:20.056 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:20.056 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:20.057 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:20.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:20.120 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:20.117+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:20.203 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:20.204 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:20.204 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:20.204 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:20.204 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:20.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:20.220 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:20.218+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:20.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:20.259 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:29:20.321 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:20.319+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7662 4.12 4:48c6f7bb:::rbd_data.11a91a08d9ca.000000000000005d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.478294237999997, 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018789818, 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.331 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7662 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:20.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849815+0000", 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5700000000000002e-07 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849819+0000", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5669999999999999e-06 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849828+0000", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1749999999999994e-06 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849832+0000", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849862+0000", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0332999999999999e-05 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.849910+0000", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7629000000000001e-05 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.851378+0000", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001468277 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868486+0000", 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017108116 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868525+0000", 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909000000000003e-05 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868565+0000", 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9218000000000003e-05 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.868604+0000", 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9926999999999999e-05 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.625594116000002, 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:20.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:20.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.604530208, 2024-09-16T13:29:20.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:20.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8593 4.2 4:472b5451:::rbd_data.11a91a08d9ca.000000000000003b:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.300457170000001, 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.020939530000000001, 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.348 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8593 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027652+0000", 2024-09-16T13:29:20.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027653+0000", 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0400000000000005e-07 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027657+0000", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1389999999999997e-06 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027668+0000", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1327e-05 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027673+0000", 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3510000000000001e-06 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027703+0000", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9434e-05 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.027789+0000", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6557000000000005e-05 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.029885+0000", 2024-09-16T13:29:20.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020960179999999998 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048420+0000", 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018534511 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048532+0000", 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112179 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048568+0000", 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5568000000000002e-05 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.048591+0000", 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3541999999999999e-05 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:20.354 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.697559947999999, 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:20.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:20.360 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25662 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.68868337, 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034733742999999997, 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639425+0000", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639426+0000", 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:29:20.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639430+0000", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639444+0000", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.358e-05 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.639451+0000", 2024-09-16T13:29:20.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674020+0000", 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034569732999999998 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674031+0000", 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.094e-05 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674159+0000", 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000127933 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25742 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.528689159999999, 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036959516999999997, 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25742 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799420+0000", 2024-09-16T13:29:20.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799421+0000", 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.282e-06 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799426+0000", 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4720000000000003e-06 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799438+0000", 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1243000000000001e-05 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799446+0000", 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799478+0000", 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2536000000000002e-05 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799524+0000", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5874000000000002e-05 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.802791+0000", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032668240000000002 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836288+0000", 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033496676000000003 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836344+0000", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5986999999999998e-05 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836364+0000", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9976e-05 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836379+0000", 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5633000000000001e-05 2024-09-16T13:29:20.370 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25743 4.f 4:f18d2a91:::rbd_data.11a926bfcc53.0000000000000006:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.528656130999998, 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036885173, 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.371 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25743 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799453+0000", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8000000000000002e-07 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799455+0000", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4080000000000001e-06 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799459+0000", 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5769999999999998e-06 2024-09-16T13:29:20.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799461+0000", 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2500000000000001e-06 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799499+0000", 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8022999999999998e-05 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799545+0000", 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5566999999999999e-05 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.799906+0000", 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000360935 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.801157+0000", 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001250962 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836259+0000", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035102510000000003 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836311+0000", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1334e-05 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836338+0000", 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7327000000000001e-05 2024-09-16T13:29:20.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:25742 4.7 e48/32 ondisk, result = 0)", 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.517292236999999, 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025474225999999999, 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810817+0000", 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810818+0000", 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.003e-06 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810824+0000", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7429999999999996e-06 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810839+0000", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4554e-05 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.810846+0000", 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9759999999999998e-06 2024-09-16T13:29:20.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836270+0000", 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025424420999999999 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836278+0000", 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0269999999999995e-06 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.836291+0000", 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2502e-05 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:28298 4.7 4:e7bc7339:::rbd_data.11a9320547a2.0000000000000003:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.028053016000001, 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033840752000000002, 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 28298 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300056+0000", 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300057+0000", 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:29:20.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300062+0000", 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2820000000000004e-06 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300075+0000", 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2975e-05 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300081+0000", 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9830000000000001e-06 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300108+0000", 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7155999999999999e-05 2024-09-16T13:29:20.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.300156+0000", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7311000000000002e-05 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.302362+0000", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002206253 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333731+0000", 2024-09-16T13:29:20.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031369085999999997 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333794+0000", 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2712999999999994e-05 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333852+0000", 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8013999999999997e-05 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333897+0000", 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4728999999999998e-05 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:28298 4.7 e51/32 ondisk, result = 0)", 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 29.019644891999999, 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025439566, 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.389 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308464+0000", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308465+0000", 2024-09-16T13:29:20.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308472+0000", 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0480000000000001e-06 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308494+0000", 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1712000000000002e-05 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.308504+0000", 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4509999999999998e-06 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333763+0000", 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025259193999999999 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333785+0000", 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2497000000000002e-05 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:51.333904+0000", 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118263 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51133 4.4 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head [call rbd.sparse_copyup in=1048616b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.014132406, 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019127457, 2024-09-16T13:29:20.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51133 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313976+0000", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.313991+0000", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4323e-05 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314605+0000", 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00061453400000000002 2024-09-16T13:29:20.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314617+0000", 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1199000000000001e-05 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314622+0000", 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2360000000000004e-06 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314665+0000", 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3058e-05 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.314816+0000", 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000150784 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315045+0000", 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022899899999999999 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.325487+0000", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010442475999999999 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333017+0000", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007529488 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333065+0000", 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8878000000000002e-05 2024-09-16T13:29:20.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333104+0000", 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8482000000000002e-05 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:58463 4.12 4:4bde84e7:::rbd_data.11a99fc6eff7.0000000000000217:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:14.706462+0000", 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.6216465419999997, 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019433148000000001, 2024-09-16T13:29:20.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 58463 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706462+0000", 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706462+0000", 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706463+0000", 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.035e-06 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706469+0000", 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2360000000000004e-06 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706479+0000", 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0814e-05 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706485+0000", 2024-09-16T13:29:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7250000000000002e-06 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706521+0000", 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5859999999999999e-05 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.706609+0000", 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7711000000000001e-05 2024-09-16T13:29:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.707030+0000", 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00042153900000000002 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.714373+0000", 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073431160000000002 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.725833+0000", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011459934 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.725870+0000", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6612000000000001e-05 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.725895+0000", 2024-09-16T13:29:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5565999999999999e-05 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:58476 4.2 4:43d8e2f4:::rbd_data.11a99fc6eff7.0000000000000098:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:14.745059+0000", 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.5830494389999998, 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024981419000000001, 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 58476 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745059+0000", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745059+0000", 2024-09-16T13:29:20.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745061+0000", 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745066+0000", 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7509999999999997e-06 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745079+0000", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2522e-05 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745084+0000", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6059999999999996e-06 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745119+0000", 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5221e-05 2024-09-16T13:29:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.745205+0000", 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6132999999999997e-05 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.746862+0000", 2024-09-16T13:29:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001656699 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.769934+0000", 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023072160000000001 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.769974+0000", 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0599999999999998e-05 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.770014+0000", 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9133e-05 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:14.770041+0000", 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7302999999999999e-05 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:59020 4.7 4:e757cc69:::rbd_data.11a99fc6eff7.00000000000000ca:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:15.385194+0000", 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9429148879999998, 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035351532999999997, 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 59020 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385194+0000", 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385194+0000", 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385196+0000", 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7349999999999999e-06 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385204+0000", 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2109999999999998e-06 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385221+0000", 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6923e-05 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385227+0000", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3060000000000003e-06 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385273+0000", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6161000000000001e-05 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.385376+0000", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102235 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.386858+0000", 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001482674 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420250+0000", 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033392257000000002 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420481+0000", 2024-09-16T13:29:20.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000230363 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420519+0000", 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7874999999999999e-05 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420545+0000", 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6792999999999999e-05 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:59021 4.12 4:4f731a1f:::rbd_data.11a99fc6eff7.00000000000000cb:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:15.389858+0000", 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9382507789999996, 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034359945000000003, 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 59021 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389858+0000", 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389858+0000", 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389859+0000", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0500000000000002e-07 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389863+0000", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9580000000000001e-06 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389874+0000", 2024-09-16T13:29:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1619e-05 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389880+0000", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.389924+0000", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3782999999999997e-05 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390067+0000", 2024-09-16T13:29:20.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014345300000000001 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.392201+0000", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021334269999999998 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424151+0000", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031950195000000001 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424158+0000", 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.002e-06 2024-09-16T13:29:20.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424192+0000", 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3970000000000002e-05 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424218+0000", 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6285e-05 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:59020 4.7 e74/32 ondisk, result = 0)", 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:15.390919+0000", 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9371895260000001, 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029333997000000001, 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390919+0000", 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390919+0000", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390920+0000", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1200000000000002e-07 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390923+0000", 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3550000000000001e-06 2024-09-16T13:29:20.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390933+0000", 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6430000000000007e-06 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.390938+0000", 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5789999999999997e-06 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420229+0000", 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029291860999999999 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420236+0000", 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6329999999999999e-06 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.420253+0000", 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7214000000000002e-05 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:59021 4.12 e74/32 ondisk, result = 0)", 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:15.397729+0000", 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.9303797469999999, 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026423495000000002, 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397729+0000", 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397729+0000", 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397730+0000", 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2550000000000001e-06 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397736+0000", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3260000000000003e-06 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397750+0000", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.45e-05 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.397758+0000", 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8120000000000008e-06 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424140+0000", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026382211999999999 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424147+0000", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4489999999999996e-06 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:15.424153+0000", 2024-09-16T13:29:20.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:61269 4.7 4:e7c53406:::rbd_data.11a911a8a02a.00000000000000da:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:18.713043+0000", 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.615065516, 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.02144623, 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 61269 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713043+0000", 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713043+0000", 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713044+0000", 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4199999999999995e-07 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713049+0000", 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4850000000000003e-06 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713058+0000", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9700000000000005e-06 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713063+0000", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2510000000000002e-06 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713106+0000", 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3211999999999999e-05 2024-09-16T13:29:20.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713188+0000", 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2318999999999996e-05 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.714587+0000", 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001398748 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734429+0000", 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019842031999999999 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734434+0000", 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.057e-06 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734468+0000", 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3624999999999997e-05 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734490+0000", 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1789000000000001e-05 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:61272 4.7 4:e4f29d31:::rbd_data.11a911a8a02a.00000000000000dd:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:18.713350+0000", 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.614759313, 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021224330999999999, 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:20.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 61272 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713350+0000", 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:20.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713350+0000", 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713350+0000", 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7400000000000003e-07 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713354+0000", 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0940000000000001e-06 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713362+0000", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.713366+0000", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8290000000000001e-06 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.727297+0000", 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013930822000000001 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.727406+0000", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109479 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.728324+0000", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00091757500000000001 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734497+0000", 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061733459999999997 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734518+0000", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0786000000000001e-05 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734544+0000", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5692999999999999e-05 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:18.734574+0000", 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0179999999999999e-05 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.444 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:20.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:20.445 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:20.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:20.447 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone25 from 0xdd1a3a9 (overlap 0xe2eff8) to 0xb58cc6 2024-09-16T13:29:20.447 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:20.448 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:20.448 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:20.449 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:20.449 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:20.449 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:20.449 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:20.443+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:20.453 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:20.470 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:20.490 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:20.490 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:20.507 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:20.507 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:20.547 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:20.545+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:20.560 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:20.560 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:20.563 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-09-16T13:29:20.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:20.592 INFO:teuthology.orchestra.run.smithi076.stdout:3353 writesame 0x548d87e thru 0x54953ea data_size 0x1(0x7b6d bytes) 2024-09-16T13:29:20.642 INFO:teuthology.orchestra.run.smithi076.stdout:3354 write 0xdb00a84 thru 0xdb1071f (0xfc9c bytes) 2024-09-16T13:29:20.648 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:20.645+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:20.659 INFO:teuthology.orchestra.run.smithi076.stdout:3355 trunc from 0xdd1a3a9 to 0x87fd43a 2024-09-16T13:29:20.686 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:20.686 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:20.688 INFO:teuthology.orchestra.run.smithi076.stdout:3358 write 0x8585ecf thru 0x858d3ff (0x7531 bytes) 2024-09-16T13:29:20.689 INFO:teuthology.orchestra.run.smithi076.stdout:3359 punch from 0x7823796 to 0x782401d, (0x887 bytes) 2024-09-16T13:29:20.690 INFO:teuthology.orchestra.run.smithi076.stdout:3360 read 0x2bf2ec8 thru 0x2bfdfc8 (0xb101 bytes) 2024-09-16T13:29:20.690 INFO:teuthology.orchestra.run.smithi076.stdout:3361 read 0x3632472 thru 0x3635ba2 (0x3731 bytes) 2024-09-16T13:29:20.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:3362 writesame 0xc834567 thru 0xc8360f9 data_size 0xd(0x1b93 bytes) 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:3363 read 0x4af83f5 thru 0x4afb141 (0x2d4d bytes) 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:3364 punch from 0x21e7d10 to 0x21ea5cf, (0x28bf bytes) 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:3365 read 0x7d844ec thru 0x7d87bd8 (0x36ed bytes) 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:20.701 INFO:teuthology.orchestra.run.smithi076.stdout:3367 write 0x6364774 thru 0x637405c (0xf8e9 bytes) 2024-09-16T13:29:20.711 INFO:teuthology.orchestra.run.smithi076.stdout:3368 read 0x6be8cec thru 0x6bf1e2b (0x9140 bytes) 2024-09-16T13:29:20.711 INFO:teuthology.orchestra.run.smithi076.stdout:3369 write 0x8dedff3 thru 0x8df77ed (0x97fb bytes) 2024-09-16T13:29:20.724 INFO:teuthology.orchestra.run.smithi076.stdout:3370 trunc from 0xc8360fa to 0x96abfdd 2024-09-16T13:29:20.730 INFO:teuthology.orchestra.run.smithi076.stdout:3371 write 0x1b56dc7 thru 0x1b624d6 (0xb710 bytes) 2024-09-16T13:29:20.747 INFO:teuthology.orchestra.run.smithi076.stdout:3373 trunc from 0x96abfdd to 0x3c5d9ce 2024-09-16T13:29:20.748 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:20.746+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:20.774 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:20.775 INFO:teuthology.orchestra.run.smithi076.stdout:3378 punch from 0x10e72d0 to 0x10e8821, (0x1551 bytes) 2024-09-16T13:29:20.790 INFO:teuthology.orchestra.run.smithi076.stdout:3379 writesame 0xc55657e thru 0xc55bc03 data_size 0x19(0x5686 bytes) 2024-09-16T13:29:20.790 INFO:teuthology.orchestra.run.smithi076.stdout:3381 clone 27 order 21 su 262144 sc 7 2024-09-16T13:29:20.816 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:20.816 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:20.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:20.866 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:20.863+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:20.951 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:20.949+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:20.963 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:20.963 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:20.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:21.051 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:21.048+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:21.069 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.069 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:21.162 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:21.159+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:21.216 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.216 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:21.269 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:21.267+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:21.329 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.329 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:21.353 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:21.350+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:21.363 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone26 from 0xc55bc04 (overlap 0x3c5d9ce) to 0x3654406 2024-09-16T13:29:21.454 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:21.452+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:21.481 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.481 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:21.562 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-09-16T13:29:21.574 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:21.572+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:21.588 INFO:teuthology.orchestra.run.smithi076.stdout:3382 punch from 0x46f71ce to 0x4700a90, (0x98c2 bytes) 2024-09-16T13:29:21.592 INFO:teuthology.orchestra.run.smithi076.stdout:truncating to largest ever: 0xee23de6 2024-09-16T13:29:21.595 INFO:teuthology.orchestra.run.smithi076.stdout:3383 trunc from 0xc55bc04 to 0xee23de6 2024-09-16T13:29:21.600 INFO:teuthology.orchestra.run.smithi076.stdout:3386 read 0x2f31030 thru 0x2f386dc (0x76ad bytes) 2024-09-16T13:29:21.600 INFO:teuthology.orchestra.run.smithi076.stdout:3389 punch from 0xc04361a to 0xc044a28, (0x140e bytes) 2024-09-16T13:29:21.601 INFO:teuthology.orchestra.run.smithi076.stdout:3390 read 0x9934617 thru 0x994226f (0xdc59 bytes) 2024-09-16T13:29:21.601 INFO:teuthology.orchestra.run.smithi076.stdout:3391 read 0x2425b92 thru 0x242b32d (0x579c bytes) 2024-09-16T13:29:21.601 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:21.608 INFO:teuthology.orchestra.run.smithi076.stdout:3394 trunc from 0xee23de6 to 0x8fd21e8 2024-09-16T13:29:21.626 INFO:teuthology.orchestra.run.smithi076.stdout:3395 read 0x4a0e4e9 thru 0x4a10e81 (0x2999 bytes) 2024-09-16T13:29:21.629 INFO:teuthology.orchestra.run.smithi076.stdout:3396 trunc from 0x8fd21e8 to 0xbe4eb07 2024-09-16T13:29:21.632 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.632 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.634 INFO:teuthology.orchestra.run.smithi076.stdout:3397 writesame 0x1659c0d thru 0x165af7c data_size 0x8(0x1370 bytes) 2024-09-16T13:29:21.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:21.642 INFO:teuthology.orchestra.run.smithi076.stdout:3398 read 0xa1eb6fd thru 0xa1f736a (0xbc6e bytes) 2024-09-16T13:29:21.642 INFO:teuthology.orchestra.run.smithi076.stdout:3399 writesame 0x1b30266 thru 0x1b3e092 data_size 0x1(0xde2d bytes) 2024-09-16T13:29:21.684 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:21.681+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:21.754 INFO:teuthology.orchestra.run.smithi076.stdout:3404 read 0x865e70a thru 0x8667b87 (0x947e bytes) 2024-09-16T13:29:21.757 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:21.754+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:21.760 INFO:teuthology.orchestra.run.smithi076.stdout:3405 writesame 0xcbdf50c thru 0xcbe953a data_size 0x1(0xa02f bytes) 2024-09-16T13:29:21.769 INFO:teuthology.orchestra.run.smithi076.stdout:3406 trunc from 0xcbe953b to 0x58bc0d1 2024-09-16T13:29:21.789 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.789 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.790 INFO:teuthology.orchestra.run.smithi076.stdout:3407 read 0x1cfc809 thru 0x1d04f6b (0x8763 bytes) 2024-09-16T13:29:21.792 INFO:teuthology.orchestra.run.smithi076.stdout:3409 trunc from 0x58bc0d1 to 0x3eea94a 2024-09-16T13:29:21.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:21.829 INFO:teuthology.orchestra.run.smithi076.stdout:3410 write 0xb5ce2fc thru 0xb5cf0ff (0xe04 bytes) 2024-09-16T13:29:21.831 INFO:teuthology.orchestra.run.smithi076.stdout:3411 write 0x4caf439 thru 0x4cb9dc0 (0xa988 bytes) 2024-09-16T13:29:21.835 INFO:teuthology.orchestra.run.smithi076.stdout:3412 punch from 0x732343d to 0x732b38c, (0x7f4f bytes) 2024-09-16T13:29:21.836 INFO:teuthology.orchestra.run.smithi076.stdout:3414 read 0x43580b0 thru 0x435bd3d (0x3c8e bytes) 2024-09-16T13:29:21.845 INFO:teuthology.orchestra.run.smithi076.stdout:3415 trunc from 0xb5cf100 to 0x39ea8f1 2024-09-16T13:29:21.855 INFO:teuthology.orchestra.run.smithi076.stdout:3418 punch from 0x2b297b5 to 0x2b2f9df, (0x622a bytes) 2024-09-16T13:29:21.857 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:21.856+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:21.870 INFO:teuthology.orchestra.run.smithi076.stdout:3422 write 0xbcfc40d thru 0xbd0668a (0xa27e bytes) 2024-09-16T13:29:21.874 INFO:teuthology.orchestra.run.smithi076.stdout:3423 write 0x9a5eebb thru 0x9a6b594 (0xc6da bytes) 2024-09-16T13:29:21.878 INFO:teuthology.orchestra.run.smithi076.stdout:3424 punch from 0x7bea81f to 0x7bf2fa9, (0x878a bytes) 2024-09-16T13:29:21.879 INFO:teuthology.orchestra.run.smithi076.stdout:3425 writesame 0xa809fe3 thru 0xa80bf8e data_size 0x2(0x1fac bytes) 2024-09-16T13:29:21.882 INFO:teuthology.orchestra.run.smithi076.stdout:3426 read 0x8e2ea6b thru 0x8e3bebe (0xd454 bytes) 2024-09-16T13:29:21.882 INFO:teuthology.orchestra.run.smithi076.stdout:3428 write 0x1ad3a72 thru 0x1ad6945 (0x2ed4 bytes) 2024-09-16T13:29:21.890 INFO:teuthology.orchestra.run.smithi076.stdout:3430 write 0x94c2ff5 thru 0x94c9ce9 (0x6cf5 bytes) 2024-09-16T13:29:21.893 INFO:teuthology.orchestra.run.smithi076.stdout:3431 writesame 0xa93b200 thru 0xa940f4c data_size 0x11(0x5d4d bytes) 2024-09-16T13:29:21.897 INFO:teuthology.orchestra.run.smithi076.stdout:3432 read 0x1c005b8 thru 0x1c0bf7d (0xb9c6 bytes) 2024-09-16T13:29:21.898 INFO:teuthology.orchestra.run.smithi076.stdout:3433 write 0x1cb1030 thru 0x1cbc27c (0xb24d bytes) 2024-09-16T13:29:21.905 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:21.905 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:21.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:21.934 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:21.934 INFO:teuthology.orchestra.run.smithi076.stdout:3435 write 0x3cc51d0 thru 0x3cca795 (0x55c6 bytes) 2024-09-16T13:29:21.936 INFO:teuthology.orchestra.run.smithi076.stdout:3436 writesame 0x1afc18 thru 0x1b7459 data_size 0x15(0x7842 bytes) 2024-09-16T13:29:21.958 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:21.956+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:21.959 INFO:teuthology.orchestra.run.smithi076.stdout:3437 write 0x91ba881 thru 0x91c0102 (0x5882 bytes) 2024-09-16T13:29:21.963 INFO:teuthology.orchestra.run.smithi076.stdout:3438 read 0x50c61f3 thru 0x50c7d6b (0x1b79 bytes) 2024-09-16T13:29:21.969 INFO:teuthology.orchestra.run.smithi076.stdout:3439 trunc from 0xbd0668b to 0x66c9eca 2024-09-16T13:29:21.979 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:21.979 INFO:teuthology.orchestra.run.smithi076.stdout:3443 writesame 0x1f1715e thru 0x1f19ed5 data_size 0x1e(0x2d78 bytes) 2024-09-16T13:29:21.991 INFO:teuthology.orchestra.run.smithi076.stdout:3444 read 0x5acf69b thru 0x5ad2347 (0x2cad bytes) 2024-09-16T13:29:21.991 INFO:teuthology.orchestra.run.smithi076.stdout:3446 read 0x3159252 thru 0x315944f (0x1fe bytes) 2024-09-16T13:29:22.006 INFO:teuthology.orchestra.run.smithi076.stdout:3448 writesame 0xe125a09 thru 0xe12fe20 data_size 0x8(0xa418 bytes) 2024-09-16T13:29:22.006 INFO:teuthology.orchestra.run.smithi076.stdout:3450 read 0x31a35c0 thru 0x31a8b6e (0x55af bytes) 2024-09-16T13:29:22.006 INFO:teuthology.orchestra.run.smithi076.stdout:3451 writesame 0x94bafed thru 0x94c90b4 data_size 0x4(0xe0c8 bytes) 2024-09-16T13:29:22.014 INFO:teuthology.orchestra.run.smithi076.stdout:3452 write 0x59a4aae thru 0x59abc7c (0x71cf bytes) 2024-09-16T13:29:22.019 INFO:teuthology.orchestra.run.smithi076.stdout:3453 writesame 0x6fbf4df thru 0x6fcc66c data_size 0x3(0xd18e bytes) 2024-09-16T13:29:22.033 INFO:teuthology.orchestra.run.smithi076.stdout:3454 write 0xe452d87 thru 0xe46029d (0xd517 bytes) 2024-09-16T13:29:22.035 INFO:teuthology.orchestra.run.smithi076.stdout:3455 writesame 0xcf4b3d1 thru 0xcf51757 data_size 0x39(0x6387 bytes) 2024-09-16T13:29:22.038 INFO:teuthology.orchestra.run.smithi076.stdout:3456 read 0x585886e thru 0x585f5fb (0x6d8e bytes) 2024-09-16T13:29:22.040 INFO:teuthology.orchestra.run.smithi076.stdout:3457 trunc from 0xe46029e to 0xd299d39 2024-09-16T13:29:22.048 INFO:teuthology.orchestra.run.smithi076.stdout:3460 writesame 0x157b066 thru 0x1587d7d data_size 0x1(0xcd18 bytes) 2024-09-16T13:29:22.058 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:22.057+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:22.063 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.063 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:22.106 INFO:teuthology.orchestra.run.smithi076.stdout:3461 read 0xcdfe8d8 thru 0xce02b3b (0x4264 bytes) 2024-09-16T13:29:22.106 INFO:teuthology.orchestra.run.smithi076.stdout:3462 write 0x91127d0 thru 0x911c3a7 (0x9bd8 bytes) 2024-09-16T13:29:22.108 INFO:teuthology.orchestra.run.smithi076.stdout:3463 read 0x9904a48 thru 0x9905809 (0xdc2 bytes) 2024-09-16T13:29:22.116 INFO:teuthology.orchestra.run.smithi076.stdout:3464 trunc from 0xd299d39 to 0x6fed38f 2024-09-16T13:29:22.140 INFO:teuthology.orchestra.run.smithi076.stdout:3465 write 0xe42952e thru 0xe42a28b (0xd5e bytes) 2024-09-16T13:29:22.142 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:22.142 INFO:teuthology.orchestra.run.smithi076.stdout:3467 punch from 0x7d7e0a3 to 0x7d7f728, (0x1685 bytes) 2024-09-16T13:29:22.142 INFO:teuthology.orchestra.run.smithi076.stdout:3468 write 0x31f4a16 thru 0x31fd7f7 (0x8de2 bytes) 2024-09-16T13:29:22.159 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:22.158+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:22.170 INFO:teuthology.orchestra.run.smithi076.stdout:3469 read 0x6edc4a9 thru 0x6ee292c (0x6484 bytes) 2024-09-16T13:29:22.170 INFO:teuthology.orchestra.run.smithi076.stdout:3471 write 0x5ada087 thru 0x5ae0098 (0x6012 bytes) 2024-09-16T13:29:22.173 INFO:teuthology.orchestra.run.smithi076.stdout:3474 read 0x49a4510 thru 0x49aa557 (0x6048 bytes) 2024-09-16T13:29:22.174 INFO:teuthology.orchestra.run.smithi076.stdout:3476 punch from 0xbfe60c0 to 0xbff2cbc, (0xcbfc bytes) 2024-09-16T13:29:22.174 INFO:teuthology.orchestra.run.smithi076.stdout:3477 read 0xe011ea8 thru 0xe01d907 (0xba60 bytes) 2024-09-16T13:29:22.174 INFO:teuthology.orchestra.run.smithi076.stdout:3478 writesame 0xc2d6cbf thru 0xc2e167e data_size 0x70(0xa9c0 bytes) 2024-09-16T13:29:22.186 INFO:teuthology.orchestra.run.smithi076.stdout:3479 trunc from 0xe42a28c to 0x5c4b758 2024-09-16T13:29:22.196 INFO:teuthology.orchestra.run.smithi076.stdout:3480 read 0x5c0f66b thru 0x5c130c2 (0x3a58 bytes) 2024-09-16T13:29:22.197 INFO:teuthology.orchestra.run.smithi076.stdout:3481 punch from 0x1ca97d9 to 0x1cb0603, (0x6e2a bytes) 2024-09-16T13:29:22.204 INFO:teuthology.orchestra.run.smithi076.stdout:3482 write 0x817fbcc thru 0x8188644 (0x8a79 bytes) 2024-09-16T13:29:22.209 INFO:teuthology.orchestra.run.smithi076.stdout:3483 read 0x41e3bf3 thru 0x41e6e7f (0x328d bytes) 2024-09-16T13:29:22.210 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:22.210 INFO:teuthology.orchestra.run.smithi076.stdout:3489 read 0x5dfb7ee thru 0x5e02156 (0x6969 bytes) 2024-09-16T13:29:22.210 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:22.210 INFO:teuthology.orchestra.run.smithi076.stdout:3491 punch from 0x6d9ba8d to 0x6d9bff3, (0x566 bytes) 2024-09-16T13:29:22.210 INFO:teuthology.orchestra.run.smithi076.stdout:3492 punch from 0x8014730 to 0x801c908, (0x81d8 bytes) 2024-09-16T13:29:22.217 INFO:teuthology.orchestra.run.smithi076.stdout:3493 write 0xa833db8 thru 0xa8341b7 (0x400 bytes) 2024-09-16T13:29:22.219 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.220 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.220 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:22.226 INFO:teuthology.orchestra.run.smithi076.stdout:3497 trunc from 0xa8341b8 to 0x53806f5 2024-09-16T13:29:22.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:22.236 INFO:teuthology.orchestra.run.smithi076.stdout:3499 read 0x3fe8463 thru 0x3ff6a14 (0xe5b2 bytes) 2024-09-16T13:29:22.239 INFO:teuthology.orchestra.run.smithi076.stdout:3501 trunc from 0x53806f5 to 0x76dfdce 2024-09-16T13:29:22.257 INFO:teuthology.orchestra.run.smithi076.stdout:3507 writesame 0xdbc02f8 thru 0xdbce61d data_size 0x5(0xe326 bytes) 2024-09-16T13:29:22.257 INFO:teuthology.orchestra.run.smithi076.stdout:3509 read 0xc95617c thru 0xc961c32 (0xbab7 bytes) 2024-09-16T13:29:22.257 INFO:teuthology.orchestra.run.smithi076.stdout:3510 write 0x3bed23a thru 0x3bee3c0 (0x1187 bytes) 2024-09-16T13:29:22.260 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:22.258+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:22.263 INFO:teuthology.orchestra.run.smithi076.stdout:3512 trunc from 0xdbce61e to 0xa440749 2024-09-16T13:29:22.268 INFO:teuthology.orchestra.run.smithi076.stdout:3513 clone 28 order 18 su 65536 sc 12 2024-09-16T13:29:22.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:29:22.361 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:22.358+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:22.368 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.368 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:22.462 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:22.459+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:22.475 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.475 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:22.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:22.564 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:22.561+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:22.582 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.582 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:22.635 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:22.651 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:22.663 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:22.662+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:22.666 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:22.680 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:22.689 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:22.689 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:22.696 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:22.696 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:22.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:22.711 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:22.727 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:22.727 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:22.742 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:22.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:22.795 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:22.792+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:22.838 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:22.838 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:22.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:22.838 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:22.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.545966734000004, 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:22.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:22.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:22.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:41 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.845 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844616536999993, 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076867799999999998, 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 41 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990551+0000", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990656+0000", 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104405 2024-09-16T13:29:22.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990665+0000", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1179999999999998e-06 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990668+0000", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991678+0000", 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00101 2024-09-16T13:29:22.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991757+0000", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9111999999999997e-05 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991897+0000", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139257 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995394+0000", 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034975929999999998 2024-09-16T13:29:22.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998186+0000", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027913299999999999 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998215+0000", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9612999999999999e-05 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998238+0000", 2024-09-16T13:29:22.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2398000000000001e-05 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844494140999998, 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.851 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:22.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844318939000004, 2024-09-16T13:29:22.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:22.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.863 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844156326000004, 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:22.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:22.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844135279, 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.870 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:22.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:22.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844047798999995, 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:22.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:22.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:22.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.844025290000005, 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.882 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:22.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:22.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:22.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.843978962999998, 2024-09-16T13:29:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:22.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:22.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.843812205999996, 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:22.894 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:22.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:22.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:22.900 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.843797441000007, 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:22.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:22.909 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.498475123999995, 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:22.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:22.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.498437445999997, 2024-09-16T13:29:22.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:22.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:22.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.921 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.498414632999996, 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:22.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.927 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.498329420000005, 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:22.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:22.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.191750495000001, 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:22.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.165842261000002, 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.938 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:22.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:22.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.097077106, 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:22.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:22.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.595534289, 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:22.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:61252 4.15 e77/70 4:ade293bf:::rbd_data.11a911a8a02a.0000000000000003:head v 77'331, mlcod=77'329)", 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.1426912250000001, 2024-09-16T13:29:22.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076711440000000004, 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692476+0000", 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692478+0000", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.502e-06 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692676+0000", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019869500000000001 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692697+0000", 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0480999999999999e-05 2024-09-16T13:29:22.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692705+0000", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4209999999999995e-06 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692762+0000", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6536000000000002e-05 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.692784+0000", 2024-09-16T13:29:22.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2266999999999999e-05 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.700105+0000", 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073212650000000004 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:18.700147+0000", 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1977000000000001e-05 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:22.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:22.956 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:22.957 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:22.955+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:22.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:23.056 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.056 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:23.056 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:23.056 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:23.057 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:23.054+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:23.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:23.158 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:23.157+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:23.203 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.203 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:23.203 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:23.203 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:23.203 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:23.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:23.260 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:23.258+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:23.268 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.320 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.151207708999998, 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.327 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.145732144, 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:23.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.385405337000002, 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:23.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.368867211999998, 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8148 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026237330000001, 2024-09-16T13:29:23.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042798758999999999, 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8148 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290636+0000", 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5900000000000003e-07 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290638+0000", 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.668e-06 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290644+0000", 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290648+0000", 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6380000000000002e-06 2024-09-16T13:29:23.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333117+0000", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042469257000000003 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333186+0000", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9132000000000002e-05 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333434+0000", 2024-09-16T13:29:23.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024816299999999998 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8157 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026220352000003, 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043153681999999999, 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8157 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:23.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290653+0000", 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7e-07 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290655+0000", 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7320000000000001e-06 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290662+0000", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290665+0000", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6890000000000002e-06 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333469+0000", 2024-09-16T13:29:23.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042804114999999997 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333518+0000", 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8687000000000003e-05 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333806+0000", 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028813999999999999 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026203762000002, 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:23.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026181416999997, 2024-09-16T13:29:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026163437000001, 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.364 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:23.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026147633000001, 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:23.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026132072000003, 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026114358000001, 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.026087775000001, 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.184533862000002, 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.987094290999998, 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:23.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:23.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:21330 4.b e45/32 ondisk, result = 0)", 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.976595543999998, 2024-09-16T13:29:23.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042512642000000003, 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340277+0000", 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340278+0000", 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340282+0000", 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4010000000000001e-06 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340293+0000", 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.340299+0000", 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382670+0000", 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042371022000000001 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382678+0000", 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382790+0000", 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112215 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.412217685000002, 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:23.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.064927727000001, 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.412 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:29:23.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.064647831, 2024-09-16T13:29:23.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:29:23.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:29:23.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.6329561150000007, 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.427 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:23.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:29:23.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.433 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:23.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:23.435 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone27 from 0xa440749 (overlap 0x39ea8f1) to 0x22bfef7 2024-09-16T13:29:23.436 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:23.432+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:23.477 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:23.477 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:23.477 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T13:29:23.477 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:29:23.477 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:29:23.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T13:29:23.535 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:23.533+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:23.580 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:23.581 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:29:23.581 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:23.584 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.584 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:23.585 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:23.585 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:23.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:23.637 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:23.635+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:23.688 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-09-16T13:29:23.738 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:23.736+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:23.767 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.767 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:23.767 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:23.767 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:23.767 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:23.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:23.839 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:23.836+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:23.941 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.234359239, 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:23.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:23.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.213295330999998, 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:23.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:23.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.306325071000003, 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.952 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:23.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:23.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:23.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72812 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.152797868, 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038465622999999997, 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72812 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784077+0000", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784078+0000", 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6080000000000001e-06 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784083+0000", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784087+0000", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.818991+0000", 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034903898000000003 2024-09-16T13:29:23.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.819060+0000", 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9875000000000004e-05 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822542+0000", 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034812910000000001 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.962 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72813 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.152783956, 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864414, 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72813 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:23.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784092+0000", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784098+0000", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5650000000000001e-06 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784102+0000", 2024-09-16T13:29:23.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0219999999999998e-06 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822570+0000", 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038468825999999998 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822652+0000", 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822734+0000", 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2639999999999995e-05 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72818 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 540672~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.150948214, 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038576821999999997, 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.967 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72818 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:23.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785928+0000", 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8300000000000001e-06 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785934+0000", 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6450000000000002e-06 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785937+0000", 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.823577+0000", 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037640060000000003 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824423+0000", 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084574599999999998 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824503+0000", 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9583e-05 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.971 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72859 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 704512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15042158799999999, 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038373760999999999, 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72859 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786453+0000", 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786455+0000", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.81e-06 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786460+0000", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4219999999999999e-06 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786463+0000", 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824530+0000", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038066804000000003 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824603+0000", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2402000000000002e-05 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824826+0000", 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022349300000000001 2024-09-16T13:29:23.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72860 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15040733100000001, 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038621214000000001, 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72860 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-07 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786469+0000", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786475+0000", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7420000000000003e-06 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786478+0000", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.613e-06 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824854+0000", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038375240999999997 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824977+0000", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012308999999999999 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825088+0000", 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111224 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15039196599999999, 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.980 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:23.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.984 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15037609699999999, 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:23.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15036105999999999, 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:23.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.993 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15034639299999999, 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:23.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:23.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.997 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15033165200000001, 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:23.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:23.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.002 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15031383300000001, 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:24.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:24.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15012942600000001, 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:24.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:24.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.011 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15011132999999999, 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:24.015 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.150090484, 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:24.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15007568900000001, 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.021 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:24.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:24.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15006108200000001, 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:24.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.15004662599999999, 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:24.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.034 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.034 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:23.938+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:24.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:24.036 INFO:teuthology.orchestra.run.smithi076.stdout:3514 read 0x37e7af0 thru 0x37edcb9 (0x61ca bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3515 read 0x88cb0f9 thru 0x88d8f69 (0xde71 bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3516 trunc from 0xa440749 to 0x6425864 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3521 writesame 0x92ca7b thru 0x938249 data_size 0x5(0xb7cf bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3522 writesame 0x8104f75 thru 0x8110f9b data_size 0x13(0xc027 bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3523 read 0x46f62e5 thru 0x46fe8a2 (0x85be bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3524 trunc from 0x8110f9c to 0xbc65e3a 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3525 write 0xc1f5969 thru 0xc1f7b11 (0x21a9 bytes) 2024-09-16T13:29:24.037 INFO:teuthology.orchestra.run.smithi076.stdout:3527 write 0xbde9128 thru 0xbdf7a30 (0xe909 bytes) 2024-09-16T13:29:24.041 INFO:teuthology.orchestra.run.smithi076.stdout:3529 trunc from 0xc1f7b12 to 0x786ea8e 2024-09-16T13:29:24.041 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:24.039+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:24.056 INFO:teuthology.orchestra.run.smithi076.stdout:3530 trunc from 0x786ea8e to 0xc74d9db 2024-09-16T13:29:24.061 INFO:teuthology.orchestra.run.smithi076.stdout:3531 punch from 0x91e6b07 to 0x91f1397, (0xa890 bytes) 2024-09-16T13:29:24.069 INFO:teuthology.orchestra.run.smithi076.stdout:3532 trunc from 0xc74d9db to 0x66df927 2024-09-16T13:29:24.074 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.074 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.074 INFO:teuthology.orchestra.run.smithi076.stdout:3535 writesame 0x432d896 thru 0x432f046 data_size 0x1(0x17b1 bytes) 2024-09-16T13:29:24.091 INFO:teuthology.orchestra.run.smithi076.stdout:3536 write 0xb13e4f2 thru 0xb14da11 (0xf520 bytes) 2024-09-16T13:29:24.096 INFO:teuthology.orchestra.run.smithi076.stdout:3538 writesame 0x4d1ba58 thru 0x4d235b9 data_size 0x11(0x7b62 bytes) 2024-09-16T13:29:24.102 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.102 INFO:teuthology.orchestra.run.smithi076.stdout:3541 read 0x32946b1 thru 0x329752a (0x2e7a bytes) 2024-09-16T13:29:24.102 INFO:teuthology.orchestra.run.smithi076.stdout:3542 punch from 0x8862672 to 0x8862eb5, (0x843 bytes) 2024-09-16T13:29:24.103 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.103 INFO:teuthology.orchestra.run.smithi076.stdout:3544 punch from 0x396e7a2 to 0x39716d3, (0x2f31 bytes) 2024-09-16T13:29:24.103 INFO:teuthology.orchestra.run.smithi076.stdout:3545 write 0x256698e thru 0x256cd4c (0x63bf bytes) 2024-09-16T13:29:24.107 INFO:teuthology.orchestra.run.smithi076.stdout:3548 read 0xace8b31 thru 0xaced93d (0x4e0d bytes) 2024-09-16T13:29:24.112 INFO:teuthology.orchestra.run.smithi076.stdout:3549 trunc from 0xb14da12 to 0x639b01d 2024-09-16T13:29:24.142 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:24.140+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:24.146 INFO:teuthology.orchestra.run.smithi076.stdout:3550 write 0xe88ff04 thru 0xe895be9 (0x5ce6 bytes) 2024-09-16T13:29:24.150 INFO:teuthology.orchestra.run.smithi076.stdout:3551 write 0x5357dc7 thru 0x5358c25 (0xe5f bytes) 2024-09-16T13:29:24.153 INFO:teuthology.orchestra.run.smithi076.stdout:3552 write 0x5cf31d9 thru 0x5cffb97 (0xc9bf bytes) 2024-09-16T13:29:24.155 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:24.155 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:24.155 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:24.155 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.157 INFO:teuthology.orchestra.run.smithi076.stdout:3555 write 0x4d0f1e0 thru 0x4d1a395 (0xb1b6 bytes) 2024-09-16T13:29:24.161 INFO:teuthology.orchestra.run.smithi076.stdout:3556 write 0x5ac5333 thru 0x5acbb46 (0x6814 bytes) 2024-09-16T13:29:24.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:24.166 INFO:teuthology.orchestra.run.smithi076.stdout:3557 write 0xb936249 thru 0xb93b02a (0x4de2 bytes) 2024-09-16T13:29:24.169 INFO:teuthology.orchestra.run.smithi076.stdout:3559 punch from 0x7636420 to 0x763feda, (0x9aba bytes) 2024-09-16T13:29:24.169 INFO:teuthology.orchestra.run.smithi076.stdout:3563 read 0x5b895d5 thru 0x5b971cd (0xdbf9 bytes) 2024-09-16T13:29:24.169 INFO:teuthology.orchestra.run.smithi076.stdout:3565 write 0xdabf8f thru 0xdb3397 (0x7409 bytes) 2024-09-16T13:29:24.178 INFO:teuthology.orchestra.run.smithi076.stdout:3567 write 0x15048f8 thru 0x1509ae0 (0x51e9 bytes) 2024-09-16T13:29:24.182 INFO:teuthology.orchestra.run.smithi076.stdout:3568 writesame 0xbf08ef6 thru 0xbf0ceb9 data_size 0x4d(0x3fc4 bytes) 2024-09-16T13:29:24.185 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.187 INFO:teuthology.orchestra.run.smithi076.stdout:3570 trunc from 0xe895bea to 0xc2ccc50 2024-09-16T13:29:24.198 INFO:teuthology.orchestra.run.smithi076.stdout:3573 write 0xa4b300b thru 0xa4b4e93 (0x1e89 bytes) 2024-09-16T13:29:24.201 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.202 INFO:teuthology.orchestra.run.smithi076.stdout:3576 read 0x5bd1aee thru 0x5bd5d0e (0x4221 bytes) 2024-09-16T13:29:24.202 INFO:teuthology.orchestra.run.smithi076.stdout:3577 punch from 0x3c86ae7 to 0x3c8f1c9, (0x86e2 bytes) 2024-09-16T13:29:24.202 INFO:teuthology.orchestra.run.smithi076.stdout:3578 punch from 0x1ded56f to 0x1df1b1a, (0x45ab bytes) 2024-09-16T13:29:24.208 INFO:teuthology.orchestra.run.smithi076.stdout:3579 trunc from 0xc2ccc50 to 0x7660848 2024-09-16T13:29:24.222 INFO:teuthology.orchestra.run.smithi076.stdout:3580 trunc from 0x7660848 to 0x3434ce3 2024-09-16T13:29:24.243 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:24.240+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:24.261 INFO:teuthology.orchestra.run.smithi076.stdout:3583 read 0x28b6dc1 thru 0x28ba0b2 (0x32f2 bytes) 2024-09-16T13:29:24.261 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.268 INFO:teuthology.orchestra.run.smithi076.stdout:3585 write 0x49bde4d thru 0x49cdc34 (0xfde8 bytes) 2024-09-16T13:29:24.274 INFO:teuthology.orchestra.run.smithi076.stdout:3586 writesame 0x2a42d31 thru 0x2a4cbd7 data_size 0x1(0x9ea7 bytes) 2024-09-16T13:29:24.289 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:24.289 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:24.289 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:24.290 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:24.290 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:24.351 INFO:teuthology.orchestra.run.smithi076.stdout:3588 writesame 0x4517671 thru 0x451e37c data_size 0xe(0x6d0c bytes) 2024-09-16T13:29:24.356 INFO:teuthology.orchestra.run.smithi076.stdout:3589 trunc from 0x49cdc35 to 0x5b24180 2024-09-16T13:29:24.362 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.362 INFO:teuthology.orchestra.run.smithi076.stdout:3591 read 0x22019bd thru 0x2206575 (0x4bb9 bytes) 2024-09-16T13:29:24.362 INFO:teuthology.orchestra.run.smithi076.stdout:3593 writesame 0x1687e9c thru 0x1695da4 data_size 0x1(0xdf09 bytes) 2024-09-16T13:29:24.407 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:24.342+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:24.439 INFO:teuthology.orchestra.run.smithi076.stdout:3594 write 0xbeb5013 thru 0xbeb97b3 (0x47a1 bytes) 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.308346307000001, 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.917528393000001, 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.279853924000001, 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50250 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.199830370000001, 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089558599999999995, 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50250 2024-09-16T13:29:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240235+0000", 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.41e-07 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240236+0000", 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.469e-06 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240242+0000", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5589999999999996e-06 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240245+0000", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0199999999999999e-06 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249091+0000", 2024-09-16T13:29:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088466229999999996 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249133+0000", 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1457e-05 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249190+0000", 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7391e-05 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50251 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.199816631000001, 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091258229999999999, 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50251 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240250+0000", 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4500000000000001e-06 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240255+0000", 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6459999999999998e-06 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240259+0000", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1889999999999999e-06 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249224+0000", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089657999999999995 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249275+0000", 2024-09-16T13:29:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0504000000000003e-05 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249374+0000", 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8917000000000002e-05 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.199802719000001, 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.199775399, 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.124821887, 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.119526610000001, 2024-09-16T13:29:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.118968052, 2024-09-16T13:29:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53277 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 335872~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446579812, 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089977589999999993, 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53277 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993487+0000", 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9889999999999999e-06 2024-09-16T13:29:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993494+0000", 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0719999999999996e-06 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993498+0000", 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9820000000000002e-06 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002382+0000", 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088848580000000007 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002428+0000", 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5374999999999998e-05 2024-09-16T13:29:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002483+0000", 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4886999999999998e-05 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53290 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446468436, 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091177910000000001, 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53290 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993597+0000", 2024-09-16T13:29:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6499999999999998e-07 2024-09-16T13:29:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993600+0000", 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2430000000000001e-06 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993609+0000", 2024-09-16T13:29:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.541e-06 2024-09-16T13:29:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993613+0000", 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5390000000000001e-06 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002526+0000", 2024-09-16T13:29:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089128339999999997 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002614+0000", 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8212000000000002e-05 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002714+0000", 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9656999999999998e-05 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53295 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446391704, 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091736779999999993, 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53295 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993674+0000", 2024-09-16T13:29:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3099999999999997e-07 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993676+0000", 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993684+0000", 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9710000000000004e-06 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993689+0000", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8099999999999997e-06 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002744+0000", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090548969999999993 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002794+0000", 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0244999999999998e-05 2024-09-16T13:29:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002847+0000", 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2241999999999999e-05 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53296 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446342763000001, 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093249630000000003, 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53296 2024-09-16T13:29:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993725+0000", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993734+0000", 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993738+0000", 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1230000000000002e-06 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002875+0000", 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091369250000000006 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002952+0000", 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7404999999999995e-05 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003047+0000", 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4820000000000004e-05 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446314549, 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446288303999999, 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.446268396000001, 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.869790393000001, 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.704442755000001, 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.688250010000001, 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:24.581 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:24.581 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.581 INFO:teuthology.orchestra.run.smithi076.stdout:3595 read 0x485ef02 thru 0x48617df (0x28de bytes) 2024-09-16T13:29:24.581 INFO:teuthology.orchestra.run.smithi076.stdout:3596 trunc from 0xbeb97b4 to 0xd4919b3 2024-09-16T13:29:24.581 INFO:teuthology.orchestra.run.smithi076.stdout:3597 write 0x88b436b thru 0x88baf40 (0x6bd6 bytes) 2024-09-16T13:29:24.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3598 read 0x2754769 thru 0x2763006 (0xe89e bytes) 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3599 punch from 0x144a033 to 0x1452985, (0x8952 bytes) 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3603 write 0xc727625 thru 0xc73430d (0xcce9 bytes) 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3605 trunc from 0xd4919b3 to 0xb086708 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3606 trunc from 0xb086708 to 0x9e771e3 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3608 trunc from 0x9e771e3 to 0xd546617 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3609 write 0x9dcb7e5 thru 0x9dd0751 (0x4f6d bytes) 2024-09-16T13:29:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:3610 read 0x990a0fe thru 0x9914c22 (0xab25 bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3611 writesame 0x1c63e44 thru 0x1c6e20d data_size 0x23(0xa3ca bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3612 writesame 0x48499de thru 0x4859231 data_size 0x2e(0xf854 bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3614 trunc from 0xd546617 to 0x99fa84d 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3615 punch from 0x4685d8 to 0x472b31, (0xa559 bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3616 writesame 0x9429569 thru 0x9439383 data_size 0x1(0xfe1b bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3617 write 0x3b900e0 thru 0x3b9d863 (0xd784 bytes) 2024-09-16T13:29:24.586 INFO:teuthology.orchestra.run.smithi076.stdout:3622 writesame 0x5d1d4b9 thru 0x5d2054c data_size 0x2(0x3094 bytes) 2024-09-16T13:29:24.586 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:24.583+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:24.588 INFO:teuthology.orchestra.run.smithi076.stdout:3623 write 0x7e9cea4 thru 0x7ea488d (0x79ea bytes) 2024-09-16T13:29:24.592 INFO:teuthology.orchestra.run.smithi076.stdout:3624 read 0x1b955c9 thru 0x1b9a400 (0x4e38 bytes) 2024-09-16T13:29:24.593 INFO:teuthology.orchestra.run.smithi076.stdout:3625 read 0x4529225 thru 0x45338a4 (0xa680 bytes) 2024-09-16T13:29:24.604 INFO:teuthology.orchestra.run.smithi076.stdout:3628 write 0xd0f5206 thru 0xd1008ee (0xb6e9 bytes) 2024-09-16T13:29:24.610 INFO:teuthology.orchestra.run.smithi076.stdout:3629 writesame 0x24e0dc0 thru 0x24e2cc2 data_size 0x1(0x1f03 bytes) 2024-09-16T13:29:24.618 INFO:teuthology.orchestra.run.smithi076.stdout:3630 write 0x291c12 thru 0x29fc80 (0xe06f bytes) 2024-09-16T13:29:24.624 INFO:teuthology.orchestra.run.smithi076.stdout:3632 punch from 0x15a21d0 to 0x15b08a9, (0xe6d9 bytes) 2024-09-16T13:29:24.625 INFO:teuthology.orchestra.run.smithi076.stdout:3633 read 0xa546fa1 thru 0xa552056 (0xb0b6 bytes) 2024-09-16T13:29:24.625 INFO:teuthology.orchestra.run.smithi076.stdout:3634 writesame 0x96296b1 thru 0x962e924 data_size 0x6(0x5274 bytes) 2024-09-16T13:29:24.631 INFO:teuthology.orchestra.run.smithi076.stdout:3635 write 0x2177463 thru 0x2179f7f (0x2b1d bytes) 2024-09-16T13:29:24.636 INFO:teuthology.orchestra.run.smithi076.stdout:3637 write 0x8e75240 thru 0x8e79de3 (0x4ba4 bytes) 2024-09-16T13:29:24.641 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.641 INFO:teuthology.orchestra.run.smithi076.stdout:3640 punch from 0x644dd4a to 0x645a236, (0xc4ec bytes) 2024-09-16T13:29:24.641 INFO:teuthology.orchestra.run.smithi076.stdout:3641 writesame 0x6091fcc thru 0x609f280 data_size 0x1(0xd2b5 bytes) 2024-09-16T13:29:24.685 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:24.683+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:24.690 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:24.691 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:24.745 INFO:teuthology.orchestra.run.smithi076.stdout:3642 read 0x5ce682e thru 0x5ce8018 (0x17eb bytes) 2024-09-16T13:29:24.745 INFO:teuthology.orchestra.run.smithi076.stdout:3644 read 0x87c61b8 thru 0x87d252e (0xc377 bytes) 2024-09-16T13:29:24.746 INFO:teuthology.orchestra.run.smithi076.stdout:3646 punch from 0x51333d7 to 0x51384b7, (0x50e0 bytes) 2024-09-16T13:29:24.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:24.761 INFO:teuthology.orchestra.run.smithi076.stdout:3648 trunc from 0xd1008ef to 0xab960 2024-09-16T13:29:24.786 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:24.784+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:24.810 INFO:teuthology.orchestra.run.smithi076.stdout:3649 write 0x1ecbb49 thru 0x1ecbd9b (0x253 bytes) 2024-09-16T13:29:24.823 INFO:teuthology.orchestra.run.smithi076.stdout:3650 trunc from 0x1ecbd9c to 0x9096761 2024-09-16T13:29:24.829 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.829 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.841 INFO:teuthology.orchestra.run.smithi076.stdout:3655 write 0xe6f4789 thru 0xe703fee (0xf866 bytes) 2024-09-16T13:29:24.847 INFO:teuthology.orchestra.run.smithi076.stdout:3659 read 0x15f4990 thru 0x1602167 (0xd7d8 bytes) 2024-09-16T13:29:24.847 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.847 INFO:teuthology.orchestra.run.smithi076.stdout:3661 punch from 0xe4b0d27 to 0xe4bf90d, (0xebe6 bytes) 2024-09-16T13:29:24.847 INFO:teuthology.orchestra.run.smithi076.stdout:3663 punch from 0x5be210f to 0x5be302c, (0xf1d bytes) 2024-09-16T13:29:24.858 INFO:teuthology.orchestra.run.smithi076.stdout:3664 trunc from 0xe703fef to 0x4b7940f 2024-09-16T13:29:24.875 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:24.875 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:24.879 INFO:teuthology.orchestra.run.smithi076.stdout:3665 writesame 0x5ea9936 thru 0x5eb451e data_size 0x1(0xabe9 bytes) 2024-09-16T13:29:24.887 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:24.885+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:24.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:24.894 INFO:teuthology.orchestra.run.smithi076.stdout:3667 write 0xc4ddf01 thru 0xc4de0df (0x1df bytes) 2024-09-16T13:29:24.896 INFO:teuthology.orchestra.run.smithi076.stdout:3668 write 0x85b008e thru 0x85ba38c (0xa2ff bytes) 2024-09-16T13:29:24.897 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:24.902 INFO:teuthology.orchestra.run.smithi076.stdout:3669 read 0xa3971fd thru 0xa3a2cb3 (0xbab7 bytes) 2024-09-16T13:29:24.902 INFO:teuthology.orchestra.run.smithi076.stdout:3670 punch from 0x4d71ddd to 0x4d7dde5, (0xc008 bytes) 2024-09-16T13:29:24.902 INFO:teuthology.orchestra.run.smithi076.stdout:3671 read 0x7884009 thru 0x78913ad (0xd3a5 bytes) 2024-09-16T13:29:24.902 INFO:teuthology.orchestra.run.smithi076.stdout:3672 read 0x9e8f429 thru 0x9e9a8f4 (0xb4cc bytes) 2024-09-16T13:29:24.902 INFO:teuthology.orchestra.run.smithi076.stdout:3673 writesame 0x1e75074 thru 0x1e7ea37 data_size 0x1a(0x99c4 bytes) 2024-09-16T13:29:24.906 INFO:teuthology.orchestra.run.smithi076.stdout:3674 write 0x505bbca thru 0x5064cb4 (0x90eb bytes) 2024-09-16T13:29:24.913 INFO:teuthology.orchestra.run.smithi076.stdout:3675 read 0x224e8c5 thru 0x22548ca (0x6006 bytes) 2024-09-16T13:29:24.913 INFO:teuthology.orchestra.run.smithi076.stdout:3676 writesame 0x964ecf8 thru 0x965239a data_size 0x1(0x36a3 bytes) 2024-09-16T13:29:24.915 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:24.915 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:24.932 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:24.932 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:24.934 INFO:teuthology.orchestra.run.smithi076.stdout:3677 writesame 0x28946fc thru 0x289efad data_size 0x8f(0xa8b2 bytes) 2024-09-16T13:29:24.939 INFO:teuthology.orchestra.run.smithi076.stdout:3678 read 0x6df0d25 thru 0x6e0025a (0xf536 bytes) 2024-09-16T13:29:24.939 INFO:teuthology.orchestra.run.smithi076.stdout:3679 punch from 0x24dbe3d to 0x24ebbb3, (0xfd76 bytes) 2024-09-16T13:29:24.939 INFO:teuthology.orchestra.run.smithi076.stdout:3680 write 0xc2030d3 thru 0xc20d07e (0x9fac bytes) 2024-09-16T13:29:24.943 INFO:teuthology.orchestra.run.smithi076.stdout:3684 write 0x35a853c thru 0x35a9ecf (0x1994 bytes) 2024-09-16T13:29:24.946 INFO:teuthology.orchestra.run.smithi076.stdout:3686 write 0x367869a thru 0x3681b10 (0x9477 bytes) 2024-09-16T13:29:24.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:24.951 INFO:teuthology.orchestra.run.smithi076.stdout:3687 write 0xa74f25c thru 0xa75039f (0x1144 bytes) 2024-09-16T13:29:24.957 INFO:teuthology.orchestra.run.smithi076.stdout:3689 trunc from 0xc4de0e0 to 0xdd29c86 2024-09-16T13:29:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:3692 write 0x70e9b5a thru 0x70f2b65 (0x900c bytes) 2024-09-16T13:29:24.966 INFO:teuthology.orchestra.run.smithi076.stdout:3693 write 0xa136249 thru 0xa13766c (0x1424 bytes) 2024-09-16T13:29:24.966 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:24.966 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:24.968 INFO:teuthology.orchestra.run.smithi076.stdout:3694 write 0x7ede1f9 thru 0x7ee585b (0x7663 bytes) 2024-09-16T13:29:24.973 INFO:teuthology.orchestra.run.smithi076.stdout:3695 read 0xbf375d thru 0xc0157f (0xde23 bytes) 2024-09-16T13:29:24.973 INFO:teuthology.orchestra.run.smithi076.stdout:3696 read 0xb10e93f thru 0xb11c350 (0xda12 bytes) 2024-09-16T13:29:24.973 INFO:teuthology.orchestra.run.smithi076.stdout:3697 write 0x771107c thru 0x7716f0b (0x5e90 bytes) 2024-09-16T13:29:24.977 INFO:teuthology.orchestra.run.smithi076.stdout:3699 read 0x734050b thru 0x73442c6 (0x3dbc bytes) 2024-09-16T13:29:24.977 INFO:teuthology.orchestra.run.smithi076.stdout:3700 read 0x18662c0 thru 0x1866c6c (0x9ad bytes) 2024-09-16T13:29:24.977 INFO:teuthology.orchestra.run.smithi076.stdout:3702 read 0xa8f3ba thru 0xa9d12d (0xdd74 bytes) 2024-09-16T13:29:24.982 INFO:teuthology.orchestra.run.smithi076.stdout:3703 trunc from 0xdd29c86 to 0xa2485b1 2024-09-16T13:29:24.986 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:24.986 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:24.988 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:24.986+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:24.994 INFO:teuthology.orchestra.run.smithi076.stdout:3704 punch from 0x73c1aa6 to 0x73ca659, (0x8bb3 bytes) 2024-09-16T13:29:24.994 INFO:teuthology.orchestra.run.smithi076.stdout:3705 read 0x65bc207 thru 0x65c1928 (0x5722 bytes) 2024-09-16T13:29:25.002 INFO:teuthology.orchestra.run.smithi076.stdout:3706 write 0xc62588f thru 0xc62a399 (0x4b0b bytes) 2024-09-16T13:29:25.003 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:25.004 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:25.005 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:25.005 INFO:teuthology.orchestra.run.smithi076.stdout:3708 clone 29 order 20 su 8192 sc 7 2024-09-16T13:29:25.009 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.009 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:25.019 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:25.019 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:25.089 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:25.087+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:25.115 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.116 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:25.191 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:25.189+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:25.221 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.221 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:25.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:29:25.330 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:25.328+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:25.330 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.331 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:25.392 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:25.391+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:25.411 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone28 from 0xc62a39a (overlap 0xab960) to 0x44a26 2024-09-16T13:29:25.446 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.447 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:25.499 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:25.498+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:25.554 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.555 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:25.593 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-09-16T13:29:25.594 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:25.592+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:25.695 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:25.694+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:25.706 INFO:teuthology.orchestra.run.smithi076.stdout:3709 read 0x128d1fd thru 0x1299a5b (0xc85f bytes) 2024-09-16T13:29:25.706 INFO:teuthology.orchestra.run.smithi076.stdout:3712 clone 30 order 24 su 131072 sc 3 2024-09-16T13:29:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.720 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:25.796 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:25.794+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:25.827 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.827 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.836 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:25.897 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:25.896+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:25.975 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:25.975 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:25.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:26.028 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:26.026+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:26.081 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.081 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:26.130 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:26.128+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:26.201 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:26.199+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:26.227 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.227 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:26.297 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:29:26.301 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:26.300+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:26.337 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.337 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:26.402 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:26.400+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:26.421 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone29 from 0xc62a39a (overlap 0xc62a39a) to 0x9e882e1 2024-09-16T13:29:26.453 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.453 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:26.507 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:26.505+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:26.570 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.570 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.579 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:26.620 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:26.618+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:26.705 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:26.704+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:26.726 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.726 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.734 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-09-16T13:29:26.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:26.739 INFO:teuthology.orchestra.run.smithi076.stdout:3713 write 0x5d9b4e0 thru 0x5d9df11 (0x2a32 bytes) 2024-09-16T13:29:26.806 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:26.805+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:26.842 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.843 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:26.866 INFO:teuthology.orchestra.run.smithi076.stdout:3714 writesame 0x37065dc thru 0x370ea23 data_size 0x44(0x8448 bytes) 2024-09-16T13:29:26.891 INFO:teuthology.orchestra.run.smithi076.stdout:3717 write 0x871cbdc thru 0x87281ea (0xb60f bytes) 2024-09-16T13:29:26.907 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:26.906+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:26.978 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:26.978 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:26.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:27.028 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:27.026+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:27.031 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:27.043 INFO:teuthology.orchestra.run.smithi076.stdout:3718 write 0xb2596e1 thru 0xb2625cc (0x8eec bytes) 2024-09-16T13:29:27.110 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:27.108+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:27.153 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:27.153 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:27.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:27.201 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:27.201 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:27.204 INFO:teuthology.orchestra.run.smithi076.stdout:3720 write 0xe256669 thru 0xe25b225 (0x4bbd bytes) 2024-09-16T13:29:27.208 INFO:teuthology.orchestra.run.smithi076.stdout:3723 punch from 0xaba8c8e to 0xabafe6f, (0x71e1 bytes) 2024-09-16T13:29:27.208 INFO:teuthology.orchestra.run.smithi076.stdout:3725 writesame 0xade55c2 thru 0xade6348 data_size 0x1(0xd87 bytes) 2024-09-16T13:29:27.211 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:27.209+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:27.211 INFO:teuthology.orchestra.run.smithi076.stdout:3726 read 0x39b381b thru 0x39bf016 (0xb7fc bytes) 2024-09-16T13:29:27.220 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:27.221 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:27.227 INFO:teuthology.orchestra.run.smithi076.stdout:3727 trunc from 0xe25b226 to 0x16a8048 2024-09-16T13:29:27.242 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:27.242 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:27.267 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:27.267 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:27.288 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:27.288 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:27.306 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:27.306 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:27.311 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:27.309+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:27.322 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:27.322 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:27.324 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:27.324 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.61909464, 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.598030731999998, 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.691060471999997, 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72812 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5375332689999999, 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038465622999999997, 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72812 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784077+0000", 2024-09-16T13:29:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784078+0000", 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6080000000000001e-06 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784083+0000", 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784087+0000", 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.818991+0000", 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034903898000000003 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.819060+0000", 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9875000000000004e-05 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822542+0000", 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034812910000000001 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72813 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5375193569999999, 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864414, 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72813 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784092+0000", 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784098+0000", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5650000000000001e-06 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784102+0000", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0219999999999998e-06 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822570+0000", 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038468825999999998 2024-09-16T13:29:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822652+0000", 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822734+0000", 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2639999999999995e-05 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72818 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 540672~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5356836149999999, 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038576821999999997, 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72818 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:29:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785928+0000", 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8300000000000001e-06 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785934+0000", 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6450000000000002e-06 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785937+0000", 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.823577+0000", 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037640060000000003 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824423+0000", 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084574599999999998 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824503+0000", 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9583e-05 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72859 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 704512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5351569889999999, 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038373760999999999, 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72859 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786452+0000", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786453+0000", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786455+0000", 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.81e-06 2024-09-16T13:29:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786460+0000", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4219999999999999e-06 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786463+0000", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824530+0000", 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038066804000000003 2024-09-16T13:29:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824603+0000", 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2402000000000002e-05 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824826+0000", 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022349300000000001 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72860 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5351427319999997, 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038621214000000001, 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72860 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-07 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786469+0000", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786475+0000", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7420000000000003e-06 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786478+0000", 2024-09-16T13:29:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.613e-06 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824854+0000", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038375240999999997 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824977+0000", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012308999999999999 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825088+0000", 2024-09-16T13:29:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111224 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5351273669999999, 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.535111498, 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5350964610000002, 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5350817939999999, 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5350670530000001, 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5350492340000002, 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5348648269999998, 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.534846731, 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5348258850000001, 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5348110899999998, 2024-09-16T13:29:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.534796483, 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.5347820269999999, 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.419 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:27.420 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.420 INFO:teuthology.orchestra.run.smithi076.stdout:3730 read 0x6d3ed8 thru 0x6d857f (0x46a8 bytes) 2024-09-16T13:29:27.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stdout:3731 write 0x5803648 thru 0x5804b66 (0x151f bytes) 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stdout:3733 read 0x38ea131 thru 0x38ed692 (0x3562 bytes) 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stdout:3734 writesame 0x31ab35d thru 0x31b1d49 data_size 0x45(0x69ed bytes) 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:27.422 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:27.423 INFO:teuthology.orchestra.run.smithi076.stdout:3736 punch from 0x434a363 to 0x43591f9, (0xee96 bytes) 2024-09-16T13:29:27.423 INFO:teuthology.orchestra.run.smithi076.stdout:3737 writesame 0x26675bb thru 0x266f3b8 data_size 0x1(0x7dfe bytes) 2024-09-16T13:29:27.424 INFO:teuthology.orchestra.run.smithi076.stdout:3738 write 0x4947ecd thru 0x495785d (0xf991 bytes) 2024-09-16T13:29:27.424 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.424 INFO:teuthology.orchestra.run.smithi076.stdout:3741 write 0x1e19a6f thru 0x1e26437 (0xc9c9 bytes) 2024-09-16T13:29:27.424 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:27.419+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:27.425 INFO:teuthology.orchestra.run.smithi076.stdout:3743 write 0xb95cbdb thru 0xb9622da (0x5700 bytes) 2024-09-16T13:29:27.428 INFO:teuthology.orchestra.run.smithi076.stdout:3744 write 0x3be0424 thru 0x3bedbc0 (0xd79d bytes) 2024-09-16T13:29:27.434 INFO:teuthology.orchestra.run.smithi076.stdout:3745 trunc from 0xb9622db to 0xdf482a9 2024-09-16T13:29:27.442 INFO:teuthology.orchestra.run.smithi076.stdout:3746 trunc from 0xdf482a9 to 0xc16e705 2024-09-16T13:29:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:3747 read 0x2f29490 thru 0x2f2f13f (0x5cb0 bytes) 2024-09-16T13:29:27.447 INFO:teuthology.orchestra.run.smithi076.stdout:3749 write 0x57ee5a7 thru 0x57f578f (0x71e9 bytes) 2024-09-16T13:29:27.449 INFO:teuthology.orchestra.run.smithi076.stdout:3751 read 0x2734ea0 thru 0x2739e63 (0x4fc4 bytes) 2024-09-16T13:29:27.450 INFO:teuthology.orchestra.run.smithi076.stdout:3752 punch from 0xbf6d114 to 0xbf6f06f, (0x1f5b bytes) 2024-09-16T13:29:27.450 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.454 INFO:teuthology.orchestra.run.smithi076.stdout:3754 trunc from 0xc16e705 to 0x8e57fd3 2024-09-16T13:29:27.466 INFO:teuthology.orchestra.run.smithi076.stdout:3755 punch from 0x2f19258 to 0x2f282c5, (0xf06d bytes) 2024-09-16T13:29:27.467 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.478 INFO:teuthology.orchestra.run.smithi076.stdout:3757 write 0xd77cc8e thru 0xd78aeeb (0xe25e bytes) 2024-09-16T13:29:27.484 INFO:teuthology.orchestra.run.smithi076.stdout:3758 trunc from 0xd78aeec to 0xbf9aca5 2024-09-16T13:29:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:3759 punch from 0x463919a to 0x464545c, (0xc2c2 bytes) 2024-09-16T13:29:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:3760 write 0x34e0d95 thru 0x34e1129 (0x395 bytes) 2024-09-16T13:29:27.501 INFO:teuthology.orchestra.run.smithi076.stdout:3762 write 0xc845453 thru 0xc84e0e6 (0x8c94 bytes) 2024-09-16T13:29:27.504 INFO:teuthology.orchestra.run.smithi076.stdout:3764 writesame 0x8156289 thru 0x81611f4 data_size 0x6d(0xaf6c bytes) 2024-09-16T13:29:27.507 INFO:teuthology.orchestra.run.smithi076.stdout:3766 read 0xdd5e1b thru 0xddfe83 (0xa069 bytes) 2024-09-16T13:29:27.508 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.508 INFO:teuthology.orchestra.run.smithi076.stdout:3769 punch from 0xa2964d5 to 0xa2a085e, (0xa389 bytes) 2024-09-16T13:29:27.508 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.521 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:27.520+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:3772 trunc from 0xc84e0e7 to 0x1af765f 2024-09-16T13:29:27.534 INFO:teuthology.orchestra.run.smithi076.stdout:3774 punch from 0x14564ac to 0x145b9e4, (0x5538 bytes) 2024-09-16T13:29:27.537 INFO:teuthology.orchestra.run.smithi076.stdout:3775 trunc from 0x1af765f to 0x405edb1 2024-09-16T13:29:27.548 INFO:teuthology.orchestra.run.smithi076.stdout:3776 trunc from 0x405edb1 to 0x7c96d67 2024-09-16T13:29:27.553 INFO:teuthology.orchestra.run.smithi076.stdout:3777 read 0xca087b thru 0xca886a (0x7ff0 bytes) 2024-09-16T13:29:27.562 INFO:teuthology.orchestra.run.smithi076.stdout:3778 trunc from 0x7c96d67 to 0x18201c6 2024-09-16T13:29:27.565 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:27.565 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:27.565 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:27.565 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:27.569 INFO:teuthology.orchestra.run.smithi076.stdout:3779 read 0xe5b387 thru 0xe64213 (0x8e8d bytes) 2024-09-16T13:29:27.569 INFO:teuthology.orchestra.run.smithi076.stdout:3782 write 0xb3b6c6 thru 0xb3f34b (0x3c86 bytes) 2024-09-16T13:29:27.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:27.586 INFO:teuthology.orchestra.run.smithi076.stdout:3784 write 0x9c6d562 thru 0x9c736ff (0x619e bytes) 2024-09-16T13:29:27.589 INFO:teuthology.orchestra.run.smithi076.stdout:3785 punch from 0x138676b to 0x1392250, (0xbae5 bytes) 2024-09-16T13:29:27.597 INFO:teuthology.orchestra.run.smithi076.stdout:3786 write 0xa449456 thru 0xa4541e7 (0xad92 bytes) 2024-09-16T13:29:27.598 INFO:teuthology.orchestra.run.smithi076.stdout:3787 writesame 0x5fb6d62 thru 0x5fb92c2 data_size 0x1(0x2561 bytes) 2024-09-16T13:29:27.618 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.618 INFO:teuthology.orchestra.run.smithi076.stdout:3790 read 0x3e62afa thru 0x3e644dd (0x19e4 bytes) 2024-09-16T13:29:27.618 INFO:teuthology.orchestra.run.smithi076.stdout:3793 writesame 0x4eaf6c1 thru 0x4ebd4ad data_size 0x1(0xdded bytes) 2024-09-16T13:29:27.622 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:27.621+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:27.686 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:27.686 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:27.686 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:27.686 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:27.686 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:27.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:27.716 INFO:teuthology.orchestra.run.smithi076.stdout:3794 read 0x339957b thru 0x33a2012 (0x8a98 bytes) 2024-09-16T13:29:27.716 INFO:teuthology.orchestra.run.smithi076.stdout:3796 write 0x668512 thru 0x6744a8 (0xbf97 bytes) 2024-09-16T13:29:27.724 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:27.722+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:27.726 INFO:teuthology.orchestra.run.smithi076.stdout:3797 trunc from 0xa4541e8 to 0x5a680d6 2024-09-16T13:29:27.737 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.742 INFO:teuthology.orchestra.run.smithi076.stdout:3800 trunc from 0x5a680d6 to 0x95cb155 2024-09-16T13:29:27.747 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.747 INFO:teuthology.orchestra.run.smithi076.stdout:3802 read 0x7b75e4a thru 0x7b80f61 (0xb118 bytes) 2024-09-16T13:29:27.747 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.754 INFO:teuthology.orchestra.run.smithi076.stdout:3804 write 0xb1d07a4 thru 0xb1d2a35 (0x2292 bytes) 2024-09-16T13:29:27.758 INFO:teuthology.orchestra.run.smithi076.stdout:3805 read 0xa8341ec thru 0xa83734d (0x3162 bytes) 2024-09-16T13:29:27.758 INFO:teuthology.orchestra.run.smithi076.stdout:3807 writesame 0x8d2db58 thru 0x8d338c9 data_size 0x1b(0x5d72 bytes) 2024-09-16T13:29:27.762 INFO:teuthology.orchestra.run.smithi076.stdout:3808 punch from 0x76beede to 0x76c667e, (0x77a0 bytes) 2024-09-16T13:29:27.762 INFO:teuthology.orchestra.run.smithi076.stdout:3810 writesame 0x4b15c50 thru 0x4b1dcd9 data_size 0x1(0x808a bytes) 2024-09-16T13:29:27.786 INFO:teuthology.orchestra.run.smithi076.stdout:3812 read 0x86f77ce thru 0x8705874 (0xe0a7 bytes) 2024-09-16T13:29:27.786 INFO:teuthology.orchestra.run.smithi076.stdout:3813 writesame 0x3b72866 thru 0x3b7452d data_size 0x8(0x1cc8 bytes) 2024-09-16T13:29:27.790 INFO:teuthology.orchestra.run.smithi076.stdout:3815 read 0x6177490 thru 0x617dce3 (0x6854 bytes) 2024-09-16T13:29:27.790 INFO:teuthology.orchestra.run.smithi076.stdout:3817 write 0x5d71c6f thru 0x5d7b2f7 (0x9689 bytes) 2024-09-16T13:29:27.793 INFO:teuthology.orchestra.run.smithi076.stdout:3818 punch from 0x45e0b69 to 0x45ec6ea, (0xbb81 bytes) 2024-09-16T13:29:27.793 INFO:teuthology.orchestra.run.smithi076.stdout:3819 write 0x4f0f699 thru 0x4f1103d (0x19a5 bytes) 2024-09-16T13:29:27.795 INFO:teuthology.orchestra.run.smithi076.stdout:3820 read 0xc4bc90 thru 0xc518ff (0x5c70 bytes) 2024-09-16T13:29:27.797 INFO:teuthology.orchestra.run.smithi076.stdout:3822 trunc from 0xb1d2a36 to 0x96b79e6 2024-09-16T13:29:27.803 INFO:teuthology.orchestra.run.smithi076.stdout:3823 write 0x6b3516d thru 0x6b37261 (0x20f5 bytes) 2024-09-16T13:29:27.815 INFO:teuthology.orchestra.run.smithi076.stdout:3826 trunc from 0x96b79e6 to 0x25fc6ca 2024-09-16T13:29:27.825 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:27.823+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.828 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.693823330999997, 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:27.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:27.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:27.834 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.303005417000001, 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:27.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.665330948000001, 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:27.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:27.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:27.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:27.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50250 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.585307394000001, 2024-09-16T13:29:27.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089558599999999995, 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50250 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240234+0000", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240235+0000", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.41e-07 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240236+0000", 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.469e-06 2024-09-16T13:29:27.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240242+0000", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5589999999999996e-06 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240245+0000", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0199999999999999e-06 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249091+0000", 2024-09-16T13:29:27.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088466229999999996 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249133+0000", 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1457e-05 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249190+0000", 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7391e-05 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50251 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.585293655000001, 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091258229999999999, 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50251 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.852 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240248+0000", 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1699999999999999e-07 2024-09-16T13:29:27.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240250+0000", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4500000000000001e-06 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240255+0000", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6459999999999998e-06 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240259+0000", 2024-09-16T13:29:27.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1889999999999999e-06 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249224+0000", 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089657999999999995 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249275+0000", 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0504000000000003e-05 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249374+0000", 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8917000000000002e-05 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.585279743000001, 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:27.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.585252423, 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:27.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.510298911, 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:27.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:27.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:27.869 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.505003634000001, 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:27.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.504445076, 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.875 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:27.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:27.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53277 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 335872~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.832056836, 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089977589999999993, 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53277 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993485+0000", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993487+0000", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9889999999999999e-06 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993494+0000", 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0719999999999996e-06 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993498+0000", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9820000000000002e-06 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002382+0000", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088848580000000007 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002428+0000", 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5374999999999998e-05 2024-09-16T13:29:27.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002483+0000", 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4886999999999998e-05 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53290 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.83194546, 2024-09-16T13:29:27.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091177910000000001, 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53290 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993596+0000", 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993597+0000", 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6499999999999998e-07 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993600+0000", 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2430000000000001e-06 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993609+0000", 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.541e-06 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993613+0000", 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5390000000000001e-06 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002526+0000", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089128339999999997 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002614+0000", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8212000000000002e-05 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002714+0000", 2024-09-16T13:29:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9656999999999998e-05 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53295 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.831868728, 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091736779999999993, 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53295 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993674+0000", 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3099999999999997e-07 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993676+0000", 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993684+0000", 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9710000000000004e-06 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993689+0000", 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8099999999999997e-06 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002744+0000", 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090548969999999993 2024-09-16T13:29:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002794+0000", 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0244999999999998e-05 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002847+0000", 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2241999999999999e-05 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53296 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.831819787000001, 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093249630000000003, 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53296 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993725+0000", 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993734+0000", 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993738+0000", 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1230000000000002e-06 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002875+0000", 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091369250000000006 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002952+0000", 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7404999999999995e-05 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003047+0000", 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4820000000000004e-05 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.831791573, 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.831765327999999, 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.831745420000001, 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.255267417000001, 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.089919779000001, 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.073727033999999, 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:27.929 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:27.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3827 write 0x5d692a2 thru 0x5d79016 (0xfd75 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3828 writesame 0x7ca6839 thru 0x7cb3b68 data_size 0x7c(0xd330 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3829 read 0x2e84692 thru 0x2e89071 (0x49e0 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3830 writesame 0x5aea370 thru 0x5aec17f data_size 0x25(0x1e10 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3831 read 0x4f6e715 thru 0x4f7dfa3 (0xf88f bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3834 punch from 0x6b2669 to 0x6bc7b2, (0xa149 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3835 read 0x2e1b8be thru 0x2e223f8 (0x6b3b bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3836 writesame 0x92395ab thru 0x9246d85 data_size 0x1(0xd7db bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:3837 write 0xa455f69 thru 0xa4604bd (0xa555 bytes) 2024-09-16T13:29:27.933 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3839 read 0x5561f21 thru 0x5570332 (0xe412 bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3840 punch from 0x71c2244 to 0x71cd2c9, (0xb085 bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3841 writesame 0x876c7f thru 0x883ab4 data_size 0x5(0xce36 bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3842 punch from 0x913e0cf to 0x913fbb1, (0x1ae2 bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3843 read 0x74d4c40 thru 0x74e2c8c (0xe04d bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3845 read 0x46199e4 thru 0x46237be (0x9ddb bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3846 write 0x95d5569 thru 0x95de867 (0x92ff bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3850 write 0x4e91683 thru 0x4ea0f5e (0xf8dc bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3852 writesame 0xc433485 thru 0xc435ed3 data_size 0x1(0x2a4f bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3853 read 0x795032e thru 0x7955f4b (0x5c1e bytes) 2024-09-16T13:29:27.934 INFO:teuthology.orchestra.run.smithi076.stdout:3854 write 0xbda5806 thru 0xbdb0847 (0xb042 bytes) 2024-09-16T13:29:27.935 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.935 INFO:teuthology.orchestra.run.smithi076.stdout:3856 writesame 0x84073a2 thru 0x840df38 data_size 0x1(0x6b97 bytes) 2024-09-16T13:29:27.935 INFO:teuthology.orchestra.run.smithi076.stdout:3857 trunc from 0xc435ed4 to 0x9350084 2024-09-16T13:29:27.935 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:27.928+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:27.936 INFO:teuthology.orchestra.run.smithi076.stdout:3858 write 0x2981693 thru 0x298be44 (0xa7b2 bytes) 2024-09-16T13:29:27.936 INFO:teuthology.orchestra.run.smithi076.stdout:3860 read 0x738e613 thru 0x7399d78 (0xb766 bytes) 2024-09-16T13:29:27.936 INFO:teuthology.orchestra.run.smithi076.stdout:3861 read 0x5a409d5 thru 0x5a4281a (0x1e46 bytes) 2024-09-16T13:29:27.936 INFO:teuthology.orchestra.run.smithi076.stdout:3862 write 0x19b72d3 thru 0x19ba10f (0x2e3d bytes) 2024-09-16T13:29:27.948 INFO:teuthology.orchestra.run.smithi076.stdout:3863 writesame 0xdf76c77 thru 0xdf792ad data_size 0x3(0x2637 bytes) 2024-09-16T13:29:27.955 INFO:teuthology.orchestra.run.smithi076.stdout:3864 trunc from 0xdf792ae to 0x793b129 2024-09-16T13:29:27.968 INFO:teuthology.orchestra.run.smithi076.stdout:3865 read 0x67bb0a thru 0x6888bc (0xcdb3 bytes) 2024-09-16T13:29:27.968 INFO:teuthology.orchestra.run.smithi076.stdout:3869 write 0x6007916 thru 0x601736f (0xfa5a bytes) 2024-09-16T13:29:27.971 INFO:teuthology.orchestra.run.smithi076.stdout:3870 read 0x4ca64ad thru 0x4cb47bf (0xe313 bytes) 2024-09-16T13:29:27.972 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.972 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:27.972 INFO:teuthology.orchestra.run.smithi076.stdout:3874 writesame 0x52b69f2 thru 0x52bfe51 data_size 0x10(0x9460 bytes) 2024-09-16T13:29:27.975 INFO:teuthology.orchestra.run.smithi076.stdout:3875 write 0xbbb17e thru 0xbc01be (0x5041 bytes) 2024-09-16T13:29:27.982 INFO:teuthology.orchestra.run.smithi076.stdout:3877 trunc from 0x793b129 to 0x3b8ef54 2024-09-16T13:29:27.993 INFO:teuthology.orchestra.run.smithi076.stdout:3878 writesame 0xbde67d thru 0xbe8cd9 data_size 0x1(0xa65d bytes) 2024-09-16T13:29:28.031 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:28.029+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:28.044 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:28.044 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:28.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.044 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.050783740000000001, 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.050816641000000003, 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.048 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:80319 4.11 e82/69)", 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:28.004565+0000", 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.038615351999999999, 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038691351999999998, 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "queued for pg", 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004565+0000", 2024-09-16T13:29:28.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004565+0000", 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004567+0000", 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6220000000000001e-06 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004571+0000", 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4959999999999998e-06 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004586+0000", 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4521999999999999e-05 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.004593+0000", 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.951e-06 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.052 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:28.052 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 2 2024-09-16T13:29:28.052 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:28.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:28.059 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:28.059 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:28.059 INFO:teuthology.orchestra.run.smithi076.stdout:3886 punch from 0x6023d0 to 0x6065da, (0x420a bytes) 2024-09-16T13:29:28.059 INFO:teuthology.orchestra.run.smithi076.stdout:3887 writesame 0x2740783 thru 0x2749711 data_size 0xd(0x8f8f bytes) 2024-09-16T13:29:28.062 INFO:teuthology.orchestra.run.smithi076.stdout:3892 read 0x4e9433 thru 0x4f2389 (0x8f57 bytes) 2024-09-16T13:29:28.062 INFO:teuthology.orchestra.run.smithi076.stdout:3894 read 0x19a8af7 thru 0x19b0eaa (0x83b4 bytes) 2024-09-16T13:29:28.063 INFO:teuthology.orchestra.run.smithi076.stdout:3895 read 0xe3f110 thru 0xe4dcb6 (0xeba7 bytes) 2024-09-16T13:29:28.063 INFO:teuthology.orchestra.run.smithi076.stdout:3897 punch from 0x296afbd to 0x296ff13, (0x4f56 bytes) 2024-09-16T13:29:28.063 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:28.075 INFO:teuthology.orchestra.run.smithi076.stdout:3899 trunc from 0x3b8ef54 to 0xde00c0c 2024-09-16T13:29:28.083 INFO:teuthology.orchestra.run.smithi076.stdout:3901 trunc from 0xde00c0c to 0xb39c02c 2024-09-16T13:29:28.088 INFO:teuthology.orchestra.run.smithi076.stdout:3904 clone 31 order 20 su 65536 sc 15 2024-09-16T13:29:28.131 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:28.130+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:28.163 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:28.163 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:28.163 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:28.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:28.232 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:28.231+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:28.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.315 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069907182000001, 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:29:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069844709000002, 2024-09-16T13:29:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:28.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:29:28.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:28.327 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069800055999998, 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:28.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069726955, 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.333 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:28.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069708449, 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:28.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:28.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:29:28.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069534496999999, 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:28.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:28.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:29:28.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.352 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.069495943, 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:28.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:29:28.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.068537248999998, 2024-09-16T13:29:28.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:28.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:29:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.068479451999998, 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:28.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:29:28.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.068385576000001, 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:28.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.068357587000001, 2024-09-16T13:29:28.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:29:28.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:29:28.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.067083561, 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:28.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:28.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.067060314999999, 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:29:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:29:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.339714166, 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:71847 4.19 4:9aaa0942:::rbd_data.11a911a8a02a.0000000000000035:1d [read 770048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.5362037900000001, 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059204723000000001, 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 71847 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9999999999999999e-07 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775954+0000", 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5090000000000001e-06 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775959+0000", 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2209999999999996e-06 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775962+0000", 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0910000000000001e-06 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.834734+0000", 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058771692 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.834803+0000", 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9303999999999999e-05 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.835157+0000", 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035360600000000002 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:71990 4.11 4:8febe650:::rbd_data.11a911a8a02a.0000000000000005:1d [read 860160~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.5313506180000003, 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059142841000000002, 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 71990 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780806+0000", 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780807+0000", 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.945e-06 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780813+0000", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4229999999999996e-06 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780816+0000", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-06 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839804+0000", 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058987824000000001 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839870+0000", 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5605999999999995e-05 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839948+0000", 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8145000000000005e-05 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.0350862640000003, 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:29:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 4.0223654340000001, 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:29:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.7993322100000002, 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:29:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.319759132, 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:29:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:29:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:28.483 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:28.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:28.485 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone30 from 0xb39c02c (overlap 0x16a8048) to 0x48800e 2024-09-16T13:29:28.485 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:28.484+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80852 4.1a 4:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b out=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:28.584417+0000", 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.00019681000000000001, 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217137, 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.586 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80852 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584417+0000", 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.587 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584417+0000", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584418+0000", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5199999999999996e-07 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584421+0000", 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8979999999999998e-06 2024-09-16T13:29:28.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584432+0000", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0604e-05 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584436+0000", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9809999999999997e-06 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584474+0000", 2024-09-16T13:29:28.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7759999999999998e-05 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584545+0000", 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1031999999999993e-05 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:28.584634+0000", 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9110000000000006e-05 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:29:28.591 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:29:28.591 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:28.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:28.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:28.591 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-16T13:29:28.591 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .001 2024-09-16T13:29:28.592 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:28.584+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:28.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:28.643 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-09-16T13:29:28.689 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:28.686+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:28.702 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:28.702 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:28.702 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:28.702 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:28.702 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:28.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:28.788 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:28.786+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:28.813 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.100058955999998, 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:28.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:28.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.090519845000003, 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.824 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.124063110000002, 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.825 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:28.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:28.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.115013261999998, 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:29:28.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:28.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:29:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:29:28.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.413651659000003, 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:28.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:28.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:28.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:28.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:28.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:28.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.403004609999996, 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:28.845 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:28.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.850 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 28.735943029000001, 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:28.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38134 4.1b e59/32 ondisk, result = 0)", 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 28.726146670999999, 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036326266000000003, 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:28.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084252+0000", 2024-09-16T13:29:28.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1039999999999999e-06 2024-09-16T13:29:28.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084256+0000", 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5820000000000002e-06 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084270+0000", 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3752e-05 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084276+0000", 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6449999999999999e-06 2024-09-16T13:29:28.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120563+0000", 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036286410999999998 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120571+0000", 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2139999999999996e-06 2024-09-16T13:29:28.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120577+0000", 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58636 4.e 4:75705afc:::rbd_data.11a99fc6eff7.00000000000001f8:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.888026962, 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036546628999999997, 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58636 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922372+0000", 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5400000000000001e-06 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922381+0000", 2024-09-16T13:29:28.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2409999999999995e-06 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922396+0000", 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5804999999999999e-05 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922403+0000", 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3199999999999996e-06 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922441+0000", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7969999999999997e-05 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922550+0000", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000109496 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.924199+0000", 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016485460000000001 2024-09-16T13:29:28.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958819+0000", 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034620214000000003 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958828+0000", 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3440000000000007e-06 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958866+0000", 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8164999999999998e-05 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958917+0000", 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0988e-05 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:58636 4.e e74/32 ondisk, result = 0)", 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.881137044000001, 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029561272999999999, 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929262+0000", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3060000000000001e-06 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929267+0000", 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6360000000000002e-06 2024-09-16T13:29:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929284+0000", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6928000000000001e-05 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929291+0000", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8050000000000007e-06 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958782+0000", 2024-09-16T13:29:28.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029490281 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958790+0000", 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7760000000000003e-06 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958822+0000", 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1541000000000002e-05 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.875 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72241 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 393216~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0401203409999997, 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026140136000000001, 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.876 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72241 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770278+0000", 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.89e-07 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770280+0000", 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770286+0000", 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9190000000000004e-06 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770289+0000", 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4759999999999998e-06 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.795326+0000", 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025036566 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.795401+0000", 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5018999999999994e-05 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796418+0000", 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010167520000000001 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72245 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 262144~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0401050740000004, 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026345113999999999, 2024-09-16T13:29:28.881 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72245 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770295+0000", 2024-09-16T13:29:28.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770301+0000", 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4999999999999996e-06 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770307+0000", 2024-09-16T13:29:28.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3299999999999998e-06 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796445+0000", 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026138443000000001 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796541+0000", 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6248000000000004e-05 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796638+0000", 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6522999999999997e-05 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72246 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0400876840000004, 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026527684999999999, 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.886 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72246 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:28.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0800000000000001e-07 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770312+0000", 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.761e-06 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770319+0000", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5570000000000001e-06 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770322+0000", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.366e-06 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796666+0000", 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026343524 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796746+0000", 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0755999999999997e-05 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796838+0000", 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1413e-05 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.890 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72247 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0400720679999997, 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026762220999999999, 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72247 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:28.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:29:28.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770328+0000", 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770334+0000", 2024-09-16T13:29:28.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9649999999999996e-06 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770337+0000", 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796864+0000", 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026526554000000001 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796943+0000", 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9411000000000003e-05 2024-09-16T13:29:28.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797088+0000", 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144716 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72248 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0400560250000002, 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026921207999999999, 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.899 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72248 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:28.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770344+0000", 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.705e-06 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770349+0000", 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770353+0000", 2024-09-16T13:29:28.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.535e-06 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797114+0000", 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026761073999999999 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797186+0000", 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2657999999999995e-05 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797263+0000", 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6553999999999999e-05 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.905 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72254 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.0400399399999998, 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02707977, 2024-09-16T13:29:28.906 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72254 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770360+0000", 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770365+0000", 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3900000000000001e-06 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770369+0000", 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2679999999999999e-06 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797289+0000", 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026920342 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797359+0000", 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9739000000000004e-05 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797438+0000", 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8798e-05 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.911 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.1683119729999998, 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:29:28.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.1511790790000003, 2024-09-16T13:29:28.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:29:28.921 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.1907901329999999, 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:28.922 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:29:28.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:28.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.927 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.1743014119999999, 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:28.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:28.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:28.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.932 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:28.932 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:28.932 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:28.933 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:28.933 INFO:teuthology.orchestra.run.smithi076.stdout:3907 trunc from 0xb39c02c to 0x313710c 2024-09-16T13:29:28.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:28.934 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:28.931+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:28.964 INFO:teuthology.orchestra.run.smithi076.stdout:3908 trunc from 0x313710c to 0x5ea38d1 2024-09-16T13:29:28.975 INFO:teuthology.orchestra.run.smithi076.stdout:3909 write 0x6c47c28 thru 0x6c55ac8 (0xdea1 bytes) 2024-09-16T13:29:28.978 INFO:teuthology.orchestra.run.smithi076.stdout:3911 read 0x4228743 thru 0x4230787 (0x8045 bytes) 2024-09-16T13:29:28.979 INFO:teuthology.orchestra.run.smithi076.stdout:3912 read 0x5790b3a thru 0x5793ce7 (0x31ae bytes) 2024-09-16T13:29:28.985 INFO:teuthology.orchestra.run.smithi076.stdout:3913 write 0x83b3c95 thru 0x83c2316 (0xe682 bytes) 2024-09-16T13:29:28.996 INFO:teuthology.orchestra.run.smithi076.stdout:3914 write 0x9d33597 thru 0x9d3ff89 (0xc9f3 bytes) 2024-09-16T13:29:29.004 INFO:teuthology.orchestra.run.smithi076.stdout:3915 writesame 0xa4a2307 thru 0xa4adc8d data_size 0x17(0xb987 bytes) 2024-09-16T13:29:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:3917 punch from 0x46178e1 to 0x461a092, (0x27b1 bytes) 2024-09-16T13:29:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:3919 read 0x44a0efc thru 0x44b0c03 (0xfd08 bytes) 2024-09-16T13:29:29.013 INFO:teuthology.orchestra.run.smithi076.stdout:3920 writesame 0xd7059b5 thru 0xd70e560 data_size 0xe(0x8bac bytes) 2024-09-16T13:29:29.013 INFO:teuthology.orchestra.run.smithi076.stdout:3921 punch from 0x6055eda to 0x6065a8b, (0xfbb1 bytes) 2024-09-16T13:29:29.013 INFO:teuthology.orchestra.run.smithi076.stdout:3922 read 0xd12af79 thru 0xd12b10f (0x197 bytes) 2024-09-16T13:29:29.020 INFO:teuthology.orchestra.run.smithi076.stdout:3923 write 0xe8e300a thru 0xe8efa06 (0xc9fd bytes) 2024-09-16T13:29:29.023 INFO:teuthology.orchestra.run.smithi076.stdout:3926 punch from 0x76dbd0 to 0x77c958, (0xed88 bytes) 2024-09-16T13:29:29.023 INFO:teuthology.orchestra.run.smithi076.stdout:3928 writesame 0x3e5b8f5 thru 0x3e665f5 data_size 0x15(0xad01 bytes) 2024-09-16T13:29:29.027 INFO:teuthology.orchestra.run.smithi076.stdout:3931 read 0x5eca002 thru 0x5ed598e (0xb98d bytes) 2024-09-16T13:29:29.027 INFO:teuthology.orchestra.run.smithi076.stdout:3932 writesame 0xcea81af thru 0xceb5b7a data_size 0x35(0xd9cc bytes) 2024-09-16T13:29:29.033 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:29.032+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:29.048 INFO:teuthology.orchestra.run.smithi076.stdout:3935 trunc from 0xe8efa07 to 0x246680f 2024-09-16T13:29:29.083 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:29.083 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:29.083 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:29.083 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:29.084 INFO:teuthology.orchestra.run.smithi076.stdout:3936 punch from 0x8c4bdd to 0x8ce3f9, (0x981c bytes) 2024-09-16T13:29:29.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:29.097 INFO:teuthology.orchestra.run.smithi076.stdout:3937 trunc from 0x246680f to 0xdc5e368 2024-09-16T13:29:29.102 INFO:teuthology.orchestra.run.smithi076.stdout:3938 read 0x2a563da thru 0x2a5c166 (0x5d8d bytes) 2024-09-16T13:29:29.102 INFO:teuthology.orchestra.run.smithi076.stdout:3939 write 0x24fde47 thru 0x2503770 (0x592a bytes) 2024-09-16T13:29:29.107 INFO:teuthology.orchestra.run.smithi076.stdout:3940 trunc from 0xdc5e368 to 0xb7d16c3 2024-09-16T13:29:29.112 INFO:teuthology.orchestra.run.smithi076.stdout:3941 write 0xac2f19b thru 0xac3aff2 (0xbe58 bytes) 2024-09-16T13:29:29.121 INFO:teuthology.orchestra.run.smithi076.stdout:3942 write 0x358bfcb thru 0x359937d (0xd3b3 bytes) 2024-09-16T13:29:29.126 INFO:teuthology.orchestra.run.smithi076.stdout:3943 write 0x2fe4f98 thru 0x2fe61d8 (0x1241 bytes) 2024-09-16T13:29:29.133 INFO:teuthology.orchestra.run.smithi076.stdout:3944 writesame 0xbe42697 thru 0xbe44e32 data_size 0x27(0x279c bytes) 2024-09-16T13:29:29.133 INFO:teuthology.orchestra.run.smithi076.stdout:3945 punch from 0x845bd5c to 0x845fd4f, (0x3ff3 bytes) 2024-09-16T13:29:29.133 INFO:teuthology.orchestra.run.smithi076.stdout:3946 writesame 0x414a21 thru 0x41d036 data_size 0x9(0x8616 bytes) 2024-09-16T13:29:29.135 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:29.134+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:29.144 INFO:teuthology.orchestra.run.smithi076.stdout:3947 writesame 0x67cc630 thru 0x67d3eb4 data_size 0x1(0x7885 bytes) 2024-09-16T13:29:29.174 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.174 INFO:teuthology.orchestra.run.smithi076.stdout:3951 write 0x4eb2ea2 thru 0x4eb44ce (0x162d bytes) 2024-09-16T13:29:29.178 INFO:teuthology.orchestra.run.smithi076.stdout:3953 read 0x8c4a89e thru 0x8c553a5 (0xab08 bytes) 2024-09-16T13:29:29.178 INFO:teuthology.orchestra.run.smithi076.stdout:3954 read 0x92c4e98 thru 0x92c6287 (0x13f0 bytes) 2024-09-16T13:29:29.178 INFO:teuthology.orchestra.run.smithi076.stdout:3957 punch from 0x6df449d to 0x6dfaab6, (0x6619 bytes) 2024-09-16T13:29:29.178 INFO:teuthology.orchestra.run.smithi076.stdout:3959 read 0x73d0a4d thru 0x73d2fba (0x256e bytes) 2024-09-16T13:29:29.178 INFO:teuthology.orchestra.run.smithi076.stdout:3960 trunc from 0xbe44e33 to 0xb480d49 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:3963 writesame 0xc808dc6 thru 0xc80c3db data_size 0x2e(0x3616 bytes) 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:3966 punch from 0x339e3e2 to 0x33ac092, (0xdcb0 bytes) 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:3967 punch from 0x168ec1a to 0x169e6f4, (0xfada bytes) 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:3971 read 0xb395df thru 0xb4016f (0x6b91 bytes) 2024-09-16T13:29:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:3972 read 0x2977e45 thru 0x297e47c (0x6638 bytes) 2024-09-16T13:29:29.196 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:29.196 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:29.196 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:29.196 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:29.196 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:29.199 INFO:teuthology.orchestra.run.smithi076.stdout:3973 trunc from 0xc80c3dc to 0x56317ff 2024-09-16T13:29:29.206 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:29.209 INFO:teuthology.orchestra.run.smithi076.stdout:3974 writesame 0x300d832 thru 0x3015467 data_size 0xd(0x7c36 bytes) 2024-09-16T13:29:29.213 INFO:teuthology.orchestra.run.smithi076.stdout:3975 read 0x2209d15 thru 0x2215cf4 (0xbfe0 bytes) 2024-09-16T13:29:29.214 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.224 INFO:teuthology.orchestra.run.smithi076.stdout:3979 write 0xa7c4375 thru 0xa7c6166 (0x1df2 bytes) 2024-09-16T13:29:29.227 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.227 INFO:teuthology.orchestra.run.smithi076.stdout:3981 read 0x181ef03 thru 0x181fdbd (0xebb bytes) 2024-09-16T13:29:29.235 INFO:teuthology.orchestra.run.smithi076.stdout:3982 trunc from 0xa7c6167 to 0x3fc2a80 2024-09-16T13:29:29.235 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:29.233+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:29.243 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.243 INFO:teuthology.orchestra.run.smithi076.stdout:3985 punch from 0x2c40fb7 to 0x2c502a0, (0xf2e9 bytes) 2024-09-16T13:29:29.244 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.244 INFO:teuthology.orchestra.run.smithi076.stdout:3988 punch from 0x2490e35 to 0x2497874, (0x6a3f bytes) 2024-09-16T13:29:29.244 INFO:teuthology.orchestra.run.smithi076.stdout:3989 read 0x2272fe2 thru 0x227d959 (0xa978 bytes) 2024-09-16T13:29:29.244 INFO:teuthology.orchestra.run.smithi076.stdout:3990 writesame 0x27fe112 thru 0x28097a7 data_size 0x1(0xb696 bytes) 2024-09-16T13:29:29.294 INFO:teuthology.orchestra.run.smithi076.stdout:3992 clone 32 order 21 su 131072 sc 9 2024-09-16T13:29:29.301 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T13:29:29.300+0000 7fb757df6640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-16T13:29:29.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd set-full-ratio 0.001 2024-09-16T13:29:29.310 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.350 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.708092301999997, 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:29.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:29.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.356 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.617561897000002, 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:29.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:29.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.590432727, 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 52.174670575, 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.367 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:29.371 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 52.104334293000001, 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.375 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.661212546999998, 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:29.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:29.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.65112328, 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.382 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:29.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 38.504334645999997, 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:29.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:29.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:29.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:29.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50162 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 581632~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.099023904999999, 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019455525000000001, 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50162 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248555+0000", 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:29.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248557+0000", 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7850000000000001e-06 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248562+0000", 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8579999999999996e-06 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248566+0000", 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.207e-06 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267854+0000", 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019288254000000001 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267938+0000", 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3678999999999996e-05 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268010+0000", 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2416000000000006e-05 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50163 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 598016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.099009054, 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019564818000000001, 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50163 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248570+0000", 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248577+0000", 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4320000000000002e-06 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248584+0000", 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248587+0000", 2024-09-16T13:29:29.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3790000000000002e-06 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268040+0000", 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019453427999999998 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268096+0000", 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5404e-05 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268134+0000", 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50164 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 614400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.098987379, 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019695615, 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.403 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50164 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:29.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248593+0000", 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9300000000000002e-06 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248599+0000", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7950000000000004e-06 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248603+0000", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268171+0000", 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019568232000000001 2024-09-16T13:29:29.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268243+0000", 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1872000000000006e-05 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268287+0000", 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3300000000000002e-05 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50165 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 630784~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.098971191, 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019831332, 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50165 2024-09-16T13:29:29.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248608+0000", 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248610+0000", 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.807e-06 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248619+0000", 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4779999999999997e-06 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248623+0000", 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0489999999999997e-06 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268313+0000", 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019690151999999999 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268379+0000", 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5792999999999998e-05 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268439+0000", 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537000000000003e-05 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50166 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.098951419999999, 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019843890999999999, 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50166 2024-09-16T13:29:29.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.65e-07 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248629+0000", 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.854e-06 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248635+0000", 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248645+0000", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7599999999999997e-06 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268466+0000", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019821442000000002 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268471+0000", 2024-09-16T13:29:29.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.775e-06 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50167 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.09892979, 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019857698999999999, 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.417 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50167 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248651+0000", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.181e-06 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248657+0000", 2024-09-16T13:29:29.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0939999999999996e-06 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248661+0000", 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268499+0000", 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019837751000000001 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268506+0000", 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7440000000000007e-06 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50168 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.098914077, 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019875693999999999, 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:29.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50168 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:29.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248665+0000", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248667+0000", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.866e-06 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248672+0000", 2024-09-16T13:29:29.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3410000000000002e-06 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248675+0000", 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.264e-06 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268536+0000", 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019860932000000001 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268540+0000", 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50929 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 327680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.050902092000001, 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019357040999999998, 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.425 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50929 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296677+0000", 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9900000000000002e-07 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296678+0000", 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.401e-06 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296683+0000", 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296686+0000", 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8949999999999998e-06 2024-09-16T13:29:29.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315967+0000", 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019280701000000001 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316000+0000", 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3549e-05 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316033+0000", 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3272999999999998e-05 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50930 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 344064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.050889749, 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019423292000000002, 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50930 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5499999999999999e-07 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296690+0000", 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.575e-06 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296696+0000", 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3460000000000001e-06 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296704+0000", 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6319999999999997e-06 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316046+0000", 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019341649999999998 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316079+0000", 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2898999999999998e-05 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316112+0000", 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2935000000000002e-05 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.434 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.372410253000002, 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:29.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:29.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:29.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.42425128, 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.442 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:29:29.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:29:29.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:29:29.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:58637 4.15 e74/70 ondisk, result = 0)", 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.414779450999999, 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026177753000000002, 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932800+0000", 2024-09-16T13:29:29.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932807+0000", 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1659999999999999e-06 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932826+0000", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9256999999999999e-05 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932835+0000", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6239999999999997e-06 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958962+0000", 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026126531000000001 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958971+0000", 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5839999999999995e-06 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958977+0000", 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1600000000000003e-06 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:29.452 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:29.453 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:29.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:29.454 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:29.364+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:29.455 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone31 from 0x3fc2a80 (overlap 0x246680f) to 0x0 2024-09-16T13:29:29.455 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:29.452+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:29.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:29.555 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:29.553+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:29.570 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:29.570 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:29.570 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:29.570 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:29.579 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:29.597 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:29.613 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:29.629 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:29.629 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:29.647 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:29.647 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:29.656 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:29.654+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:29.664 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:29.664 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:29.680 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:29.680 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:29.681 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:29.681 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:29.681 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:29.681 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:29.681 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:29.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:29.698 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:29.699 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:29.720 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:29.720 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:29.733 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-09-16T13:29:29.757 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:29.755+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:29.766 INFO:teuthology.orchestra.run.smithi076.stdout:3993 write 0xe077a26 thru 0xe086860 (0xee3b bytes) 2024-09-16T13:29:29.769 INFO:teuthology.orchestra.run.smithi076.stdout:3995 read 0xc341ef thru 0xc3461e (0x430 bytes) 2024-09-16T13:29:29.770 INFO:teuthology.orchestra.run.smithi076.stdout:3996 writesame 0xbd6cc50 thru 0xbd7014b data_size 0x2(0x34fc bytes) 2024-09-16T13:29:29.784 INFO:teuthology.orchestra.run.smithi076.stdout:3997 read 0x926ab92 thru 0x926d2da (0x2749 bytes) 2024-09-16T13:29:29.784 INFO:teuthology.orchestra.run.smithi076.stdout:3999 read 0x975160b thru 0x975fd83 (0xe779 bytes) 2024-09-16T13:29:29.785 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.785 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.785 INFO:teuthology.orchestra.run.smithi076.stdout:4002 trunc from 0xe086861 to 0xe1ce39e 2024-09-16T13:29:29.790 INFO:teuthology.orchestra.run.smithi076.stdout:4003 punch from 0x5188176 to 0x518cf25, (0x4daf bytes) 2024-09-16T13:29:29.790 INFO:teuthology.orchestra.run.smithi076.stdout:4004 read 0xcde0ad0 thru 0xcdf01f3 (0xf724 bytes) 2024-09-16T13:29:29.792 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:29.792 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:29.792 INFO:teuthology.orchestra.run.smithi076.stdout:4005 trunc from 0xe1ce39e to 0xc63877a 2024-09-16T13:29:29.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:29.802 INFO:teuthology.orchestra.run.smithi076.stdout:4006 read 0x21e2d4f thru 0x21e62f1 (0x35a3 bytes) 2024-09-16T13:29:29.802 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.802 INFO:teuthology.orchestra.run.smithi076.stdout:4009 write 0xba924a9 thru 0xba9c547 (0xa09f bytes) 2024-09-16T13:29:29.813 INFO:teuthology.orchestra.run.smithi076.stdout:4010 write 0xda7cd8d thru 0xda82a88 (0x5cfc bytes) 2024-09-16T13:29:29.819 INFO:teuthology.orchestra.run.smithi076.stdout:4011 read 0xa78ec4b thru 0xa7903c5 (0x177b bytes) 2024-09-16T13:29:29.819 INFO:teuthology.orchestra.run.smithi076.stdout:4013 read 0x3863868 thru 0x3871634 (0xddcd bytes) 2024-09-16T13:29:29.820 INFO:teuthology.orchestra.run.smithi076.stdout:4014 read 0x1cfdf62 thru 0x1d01ddf (0x3e7e bytes) 2024-09-16T13:29:29.820 INFO:teuthology.orchestra.run.smithi076.stdout:4015 writesame 0x184aa96 thru 0x184eb22 data_size 0x5(0x408d bytes) 2024-09-16T13:29:29.829 INFO:teuthology.orchestra.run.smithi076.stdout:4016 punch from 0xabc65bc to 0xabd0912, (0xa356 bytes) 2024-09-16T13:29:29.829 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.829 INFO:teuthology.orchestra.run.smithi076.stdout:4021 punch from 0x46ace99 to 0x46ae979, (0x1ae0 bytes) 2024-09-16T13:29:29.842 INFO:teuthology.orchestra.run.smithi076.stdout:4023 trunc from 0xda82a89 to 0x22a2fbf 2024-09-16T13:29:29.858 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:29.856+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:29.894 INFO:teuthology.orchestra.run.smithi076.stdout:4024 trunc from 0x22a2fbf to 0xa48e714 2024-09-16T13:29:29.900 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.900 INFO:teuthology.orchestra.run.smithi076.stdout:4026 read 0x2091a61 thru 0x2092374 (0x914 bytes) 2024-09-16T13:29:29.901 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:29.901 INFO:teuthology.orchestra.run.smithi076.stdout:4028 read 0x2b36612 thru 0x2b3f410 (0x8dff bytes) 2024-09-16T13:29:29.901 INFO:teuthology.orchestra.run.smithi076.stdout:4029 read 0xad616f thru 0xade8fa (0x878c bytes) 2024-09-16T13:29:29.906 INFO:teuthology.orchestra.run.smithi076.stdout:4030 write 0xa6cb573 thru 0xa6d9a36 (0xe4c4 bytes) 2024-09-16T13:29:29.910 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:29.911 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:29.913 INFO:teuthology.orchestra.run.smithi076.stdout:4031 trunc from 0xa6d9a37 to 0x765aac9 2024-09-16T13:29:29.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:29.922 INFO:teuthology.orchestra.run.smithi076.stdout:4032 write 0x52748c3 thru 0x5283e68 (0xf5a6 bytes) 2024-09-16T13:29:29.928 INFO:teuthology.orchestra.run.smithi076.stdout:4033 punch from 0x64168a1 to 0x6420eeb, (0xa64a bytes) 2024-09-16T13:29:29.928 INFO:teuthology.orchestra.run.smithi076.stdout:4035 read 0x245915d thru 0x2460e3a (0x7cde bytes) 2024-09-16T13:29:29.928 INFO:teuthology.orchestra.run.smithi076.stdout:4036 read 0x4391402 thru 0x4392ba4 (0x17a3 bytes) 2024-09-16T13:29:29.928 INFO:teuthology.orchestra.run.smithi076.stdout:4038 write 0x66d765b thru 0x66e5c04 (0xe5aa bytes) 2024-09-16T13:29:29.933 INFO:teuthology.orchestra.run.smithi076.stdout:4039 read 0x63793fc thru 0x637d610 (0x4215 bytes) 2024-09-16T13:29:29.933 INFO:teuthology.orchestra.run.smithi076.stdout:4040 write 0xe1be58 thru 0xe23667 (0x7810 bytes) 2024-09-16T13:29:29.956 INFO:teuthology.orchestra.run.smithi076.stdout:4041 read 0x5981f13 thru 0x59855e7 (0x36d5 bytes) 2024-09-16T13:29:29.963 INFO:teuthology.orchestra.run.smithi076.stdout:4042 write 0x8e850c5 thru 0x8e856a2 (0x5de bytes) 2024-09-16T13:29:29.963 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:29.962+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:29.965 INFO:teuthology.orchestra.run.smithi076.stdout:4044 clone 33 order 24 su 1048576 sc 6 2024-09-16T13:29:30.022 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.022 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:30.059 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:30.057+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:30.160 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:30.158+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:30.170 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.170 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:30.261 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:30.260+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:30.277 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.277 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:30.326 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-16T13:29:30.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .95 2024-09-16T13:29:30.371 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:30.370+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:30.433 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.433 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:30.462 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:30.461+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone32 from 0x8e856a3 (overlap 0x22a2fbf) to 0x14c830c 2024-09-16T13:29:30.563 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:30.562+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:30.576 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.576 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:30.621 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-09-16T13:29:30.623 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:30.623 INFO:teuthology.orchestra.run.smithi076.stdout:4047 read 0x5f8ec7c thru 0x5f9334b (0x46d0 bytes) 2024-09-16T13:29:30.625 INFO:teuthology.orchestra.run.smithi076.stdout:4049 trunc from 0x8e856a3 to 0x68d70f7 2024-09-16T13:29:30.646 INFO:teuthology.orchestra.run.smithi076.stdout:4052 read 0x3e5098c thru 0x3e5bfe6 (0xb65b bytes) 2024-09-16T13:29:30.658 INFO:teuthology.orchestra.run.smithi076.stdout:4053 write 0xa8d9e67 thru 0xa8e633a (0xc4d4 bytes) 2024-09-16T13:29:30.659 INFO:teuthology.orchestra.run.smithi076.stdout:4055 read 0x7992443 thru 0x79991c7 (0x6d85 bytes) 2024-09-16T13:29:30.659 INFO:teuthology.orchestra.run.smithi076.stdout:4057 read 0x204452 thru 0x20b466 (0x7015 bytes) 2024-09-16T13:29:30.660 INFO:teuthology.orchestra.run.smithi076.stdout:4058 write 0x543805c thru 0x543b089 (0x302e bytes) 2024-09-16T13:29:30.664 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:30.662+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:30.689 INFO:teuthology.orchestra.run.smithi076.stdout:4061 read 0x644a3a8 thru 0x644ba13 (0x166c bytes) 2024-09-16T13:29:30.690 INFO:teuthology.orchestra.run.smithi076.stdout:4064 read 0x1c8e92b thru 0x1c98caa (0xa380 bytes) 2024-09-16T13:29:30.690 INFO:teuthology.orchestra.run.smithi076.stdout:4066 punch from 0x69cfe75 to 0x69d9851, (0x99dc bytes) 2024-09-16T13:29:30.701 INFO:teuthology.orchestra.run.smithi076.stdout:4067 writesame 0xde0a489 thru 0xde0d6e7 data_size 0x1(0x325f bytes) 2024-09-16T13:29:30.701 INFO:teuthology.orchestra.run.smithi076.stdout:4068 write 0x8b9e333 thru 0x8bad8c4 (0xf592 bytes) 2024-09-16T13:29:30.705 INFO:teuthology.orchestra.run.smithi076.stdout:4069 punch from 0x670ac3b to 0x6716cf0, (0xc0b5 bytes) 2024-09-16T13:29:30.705 INFO:teuthology.orchestra.run.smithi076.stdout:4070 write 0xdc21152 thru 0xdc21657 (0x506 bytes) 2024-09-16T13:29:30.708 INFO:teuthology.orchestra.run.smithi076.stdout:4073 write 0x4478724 thru 0x447d68d (0x4f6a bytes) 2024-09-16T13:29:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.734 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.737 INFO:teuthology.orchestra.run.smithi076.stdout:4074 trunc from 0xde0d6e8 to 0xb47d437 2024-09-16T13:29:30.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:30.752 INFO:teuthology.orchestra.run.smithi076.stdout:4076 write 0x89f4184 thru 0x89f4cd8 (0xb55 bytes) 2024-09-16T13:29:30.753 INFO:teuthology.orchestra.run.smithi076.stdout:4077 write 0x4890673 thru 0x4895b56 (0x54e4 bytes) 2024-09-16T13:29:30.755 INFO:teuthology.orchestra.run.smithi076.stdout:4078 read 0x21ca01d thru 0x21cb401 (0x13e5 bytes) 2024-09-16T13:29:30.755 INFO:teuthology.orchestra.run.smithi076.stdout:4079 writesame 0x396b171 thru 0x396c4c9 data_size 0xd(0x1359 bytes) 2024-09-16T13:29:30.763 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:30.764 INFO:teuthology.orchestra.run.smithi076.stdout:4081 trunc from 0xb47d437 to 0xbcc72b7 2024-09-16T13:29:30.764 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:30.763+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:30.769 INFO:teuthology.orchestra.run.smithi076.stdout:4082 read 0x8e3f21a thru 0x8e42c6c (0x3a53 bytes) 2024-09-16T13:29:30.769 INFO:teuthology.orchestra.run.smithi076.stdout:4083 read 0x6af4f1f thru 0x6afe62e (0x9710 bytes) 2024-09-16T13:29:30.772 INFO:teuthology.orchestra.run.smithi076.stdout:4084 trunc from 0xbcc72b7 to 0x9af56b9 2024-09-16T13:29:30.780 INFO:teuthology.orchestra.run.smithi076.stdout:4085 writesame 0xb1817e thru 0xb1c19c data_size 0x7(0x401f bytes) 2024-09-16T13:29:30.805 INFO:teuthology.orchestra.run.smithi076.stdout:4086 punch from 0x6aa6b45 to 0x6aa7c83, (0x113e bytes) 2024-09-16T13:29:30.805 INFO:teuthology.orchestra.run.smithi076.stdout:4087 read 0x8e1d73a thru 0x8e23408 (0x5ccf bytes) 2024-09-16T13:29:30.805 INFO:teuthology.orchestra.run.smithi076.stdout:4088 read 0x2a93561 thru 0x2a95c48 (0x26e8 bytes) 2024-09-16T13:29:30.814 INFO:teuthology.orchestra.run.smithi076.stdout:4089 write 0xb6cacb6 thru 0xb6d131e (0x6669 bytes) 2024-09-16T13:29:30.828 INFO:teuthology.orchestra.run.smithi076.stdout:4091 trunc from 0xb6d131f to 0x8f7130 2024-09-16T13:29:30.863 INFO:teuthology.orchestra.run.smithi076.stdout:4092 writesame 0x15fe1ff thru 0x15ff60d data_size 0xd(0x140f bytes) 2024-09-16T13:29:30.865 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:30.864+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:30.867 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.867 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.877 INFO:teuthology.orchestra.run.smithi076.stdout:4093 write 0x6ce6540 thru 0x6cef8b7 (0x9378 bytes) 2024-09-16T13:29:30.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:30.880 INFO:teuthology.orchestra.run.smithi076.stdout:4094 read 0x3bd280b thru 0x3bd709c (0x4892 bytes) 2024-09-16T13:29:30.880 INFO:teuthology.orchestra.run.smithi076.stdout:4096 read 0x20649e0 thru 0x2067863 (0x2e84 bytes) 2024-09-16T13:29:30.882 INFO:teuthology.orchestra.run.smithi076.stdout:4097 trunc from 0x6cef8b8 to 0x8411434 2024-09-16T13:29:30.899 INFO:teuthology.orchestra.run.smithi076.stdout:4098 writesame 0xd3750fa thru 0xd375370 data_size 0x1(0x277 bytes) 2024-09-16T13:29:30.899 INFO:teuthology.orchestra.run.smithi076.stdout:4099 read 0xc845c90 thru 0xc849a3b (0x3dac bytes) 2024-09-16T13:29:30.900 INFO:teuthology.orchestra.run.smithi076.stdout:4100 write 0x868d8d5 thru 0x869bdcf (0xe4fb bytes) 2024-09-16T13:29:30.905 INFO:teuthology.orchestra.run.smithi076.stdout:4101 write 0x7167572 thru 0x716eb90 (0x761f bytes) 2024-09-16T13:29:30.915 INFO:teuthology.orchestra.run.smithi076.stdout:4102 write 0xe056f42 thru 0xe066984 (0xfa43 bytes) 2024-09-16T13:29:30.918 INFO:teuthology.orchestra.run.smithi076.stdout:4103 punch from 0x13c3e81 to 0x13d265e, (0xe7dd bytes) 2024-09-16T13:29:30.924 INFO:teuthology.orchestra.run.smithi076.stdout:4104 write 0xe9e069a thru 0xe9e7820 (0x7187 bytes) 2024-09-16T13:29:30.927 INFO:teuthology.orchestra.run.smithi076.stdout:4105 punch from 0x12d92e6 to 0x12dbf21, (0x2c3b bytes) 2024-09-16T13:29:30.927 INFO:teuthology.orchestra.run.smithi076.stdout:4106 write 0x5f5dcea thru 0x5f61565 (0x387c bytes) 2024-09-16T13:29:30.929 INFO:teuthology.orchestra.run.smithi076.stdout:4107 write 0x5a34855 thru 0x5a36866 (0x2012 bytes) 2024-09-16T13:29:30.930 INFO:teuthology.orchestra.run.smithi076.stdout:4108 writesame 0x17dc11f thru 0x17e3b3d data_size 0x11(0x7a1f bytes) 2024-09-16T13:29:30.933 INFO:teuthology.orchestra.run.smithi076.stdout:4110 writesame 0xe91b8a1 thru 0xe9244a3 data_size 0x1(0x8c03 bytes) 2024-09-16T13:29:30.966 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:30.964+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:30.991 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:30.991 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:30.999 INFO:teuthology.orchestra.run.smithi076.stdout:4111 writesame 0x8482998 thru 0x848ceae data_size 0x1(0xa517 bytes) 2024-09-16T13:29:31.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:31.036 INFO:teuthology.orchestra.run.smithi076.stdout:4112 punch from 0xd9e4411 to 0xd9e9620, (0x520f bytes) 2024-09-16T13:29:31.037 INFO:teuthology.orchestra.run.smithi076.stdout:4113 write 0x4c29283 thru 0x4c34047 (0xadc5 bytes) 2024-09-16T13:29:31.039 INFO:teuthology.orchestra.run.smithi076.stdout:4114 write 0x38d1471 thru 0x38dd851 (0xc3e1 bytes) 2024-09-16T13:29:31.043 INFO:teuthology.orchestra.run.smithi076.stdout:4116 writesame 0xb1e1e37 thru 0xb1ec25a data_size 0x6e(0xa424 bytes) 2024-09-16T13:29:31.046 INFO:teuthology.orchestra.run.smithi076.stdout:4117 punch from 0xc57b763 to 0xc580995, (0x5232 bytes) 2024-09-16T13:29:31.046 INFO:teuthology.orchestra.run.smithi076.stdout:4119 writesame 0x4f2fa87 thru 0x4f3ba4e data_size 0x88(0xbfc8 bytes) 2024-09-16T13:29:31.047 INFO:teuthology.orchestra.run.smithi076.stdout:4120 punch from 0x978bfad to 0x9795ad5, (0x9b28 bytes) 2024-09-16T13:29:31.048 INFO:teuthology.orchestra.run.smithi076.stdout:4121 read 0x82e5f11 thru 0x82f52d4 (0xf3c4 bytes) 2024-09-16T13:29:31.048 INFO:teuthology.orchestra.run.smithi076.stdout:4122 read 0x2e4c6ca thru 0x2e525a2 (0x5ed9 bytes) 2024-09-16T13:29:31.048 INFO:teuthology.orchestra.run.smithi076.stdout:4123 write 0x64e1fe3 thru 0x64ef829 (0xd847 bytes) 2024-09-16T13:29:31.054 INFO:teuthology.orchestra.run.smithi076.stdout:4124 read 0xc928a77 thru 0xc92e56f (0x5af9 bytes) 2024-09-16T13:29:31.054 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:31.054 INFO:teuthology.orchestra.run.smithi076.stdout:4126 write 0x758868e thru 0x7588dc5 (0x738 bytes) 2024-09-16T13:29:31.054 INFO:teuthology.orchestra.run.smithi076.stdout:4129 write 0x37f8932 thru 0x37fe681 (0x5d50 bytes) 2024-09-16T13:29:31.057 INFO:teuthology.orchestra.run.smithi076.stdout:4130 writesame 0x2ffd69b thru 0x300d143 data_size 0x7(0xfaa9 bytes) 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:4133 punch from 0x92875ba to 0x928d8a2, (0x62e8 bytes) 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:4135 punch from 0x937ecae to 0x938cc0d, (0xdf5f bytes) 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:4136 read 0xbe9fc79 thru 0xbea1468 (0x17f0 bytes) 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:31.062 INFO:teuthology.orchestra.run.smithi076.stdout:4139 read 0xa8fdba7 thru 0xa8fe7cd (0xc27 bytes) 2024-09-16T13:29:31.063 INFO:teuthology.orchestra.run.smithi076.stdout:4140 clone 34 order 21 su 131072 sc 6 2024-09-16T13:29:31.067 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:31.065+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:31.186 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:31.184+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:31.268 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:31.266+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:31.283 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.283 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:31.320 INFO:teuthology.orchestra.run.smithi076.stderr:osd set-full-ratio 0.95 2024-09-16T13:29:31.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:29:31.369 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:31.367+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:31.426 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.426 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:31.479 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:31.478+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:31.500 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone33 from 0xe9e7821 (overlap 0x8f7130) to 0x0 2024-09-16T13:29:31.571 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:31.569+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:31.574 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.574 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:31.671 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:31.670+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:31.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:31.732 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.732 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.738 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-09-16T13:29:31.742 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:31.780 INFO:teuthology.orchestra.run.smithi076.stdout:4141 read 0x493ec63 thru 0x49435b2 (0x4950 bytes) 2024-09-16T13:29:31.780 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:31.778+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:31.782 INFO:teuthology.orchestra.run.smithi076.stdout:4142 write 0xb97b4 thru 0xc29a9 (0x91f6 bytes) 2024-09-16T13:29:31.796 INFO:teuthology.orchestra.run.smithi076.stdout:4143 read 0x2e1f055 thru 0x2e1fe86 (0xe32 bytes) 2024-09-16T13:29:31.797 INFO:teuthology.orchestra.run.smithi076.stdout:4144 read 0xc9ecedf thru 0xc9fca39 (0xfb5b bytes) 2024-09-16T13:29:31.797 INFO:teuthology.orchestra.run.smithi076.stdout:4146 writesame 0xd216ae7 thru 0xd218102 data_size 0xa(0x161c bytes) 2024-09-16T13:29:31.801 INFO:teuthology.orchestra.run.smithi076.stdout:4148 punch from 0x2d19c13 to 0x2d1a1c4, (0x5b1 bytes) 2024-09-16T13:29:31.801 INFO:teuthology.orchestra.run.smithi076.stdout:4149 flatten 2024-09-16T13:29:31.874 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:31.872+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:31.909 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:31.939 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:31.939 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:31.942 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:31.942 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:31.949 INFO:teuthology.orchestra.run.smithi076.stdout:4151 trunc from 0xe9e7821 to 0x24905ea 2024-09-16T13:29:31.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:31.956 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:31.974 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:31.978 INFO:teuthology.orchestra.run.smithi076.stdout:4152 trunc from 0x24905ea to 0x7e24663 2024-09-16T13:29:31.988 INFO:teuthology.orchestra.run.smithi076.stdout:4153 write 0x89a9ffb thru 0x89ae039 (0x403f bytes) 2024-09-16T13:29:31.991 INFO:teuthology.orchestra.run.smithi076.stdout:4154 clone 35 order 23 su 4194304 sc 10 2024-09-16T13:29:31.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:32.001 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:31.999+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:32.009 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:32.009 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:32.024 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:32.024 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:32.039 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:32.040 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:32.057 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.057 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:32.057 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:32.057 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:32.107 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:32.105+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:32.176 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:32.174+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:32.204 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.204 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:32.204 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:32.205 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:32.205 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:32.278 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:32.276+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:32.312 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.312 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:32.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:32.313 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:32.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.313 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:32.313 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:32.314 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.598830247999999, 2024-09-16T13:29:32.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:32.314 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.314 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.315 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.315 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.315 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:32.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.315 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:32.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:32.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:32.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:32.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:32.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:32.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:32.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:32.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:32.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:32.330 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:32.331 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.589291137000004, 2024-09-16T13:29:32.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:32.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.332 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.332 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:32.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:32.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.622834402000002, 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.340 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.341 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:32.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:32.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:32.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:32.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:32.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:32.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:32.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25675 4.1a e48/32 ondisk, result = 0)", 2024-09-16T13:29:32.352 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.613784553999999, 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034262977, 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695385+0000", 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695386+0000", 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.074e-06 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695390+0000", 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999998e-06 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695407+0000", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6447999999999999e-05 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.695414+0000", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729564+0000", 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034150634999999999 2024-09-16T13:29:32.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729571+0000", 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5830000000000003e-06 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729647+0000", 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6515999999999997e-05 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.912422951000003, 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.357 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:32.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.901775901999997, 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:32.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:32.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.234714320999998, 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:32.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:32.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38134 4.1b e59/32 ondisk, result = 0)", 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.224917963000003, 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036326266000000003, 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084251+0000", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084252+0000", 2024-09-16T13:29:32.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1039999999999999e-06 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084256+0000", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5820000000000002e-06 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084270+0000", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3752e-05 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.084276+0000", 2024-09-16T13:29:32.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6449999999999999e-06 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120563+0000", 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036286410999999998 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120571+0000", 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2139999999999996e-06 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120577+0000", 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58636 4.e 4:75705afc:::rbd_data.11a99fc6eff7.00000000000001f8:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.386798253999999, 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036546628999999997, 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58636 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922371+0000", 2024-09-16T13:29:32.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922372+0000", 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5400000000000001e-06 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922381+0000", 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2409999999999995e-06 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922396+0000", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5804999999999999e-05 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922403+0000", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3199999999999996e-06 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922441+0000", 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7969999999999997e-05 2024-09-16T13:29:32.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.922550+0000", 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000109496 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.924199+0000", 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016485460000000001 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958819+0000", 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034620214000000003 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958828+0000", 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3440000000000007e-06 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958866+0000", 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8164999999999998e-05 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958917+0000", 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0988e-05 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:58636 4.e e74/32 ondisk, result = 0)", 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.379908336, 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029561272999999999, 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.383 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929261+0000", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929262+0000", 2024-09-16T13:29:32.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3060000000000001e-06 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929267+0000", 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6360000000000002e-06 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929284+0000", 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6928000000000001e-05 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.929291+0000", 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8050000000000007e-06 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958782+0000", 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029490281 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958790+0000", 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7760000000000003e-06 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958822+0000", 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1541000000000002e-05 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72241 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 393216~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388916330000004, 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026140136000000001, 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72241 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770277+0000", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770278+0000", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.89e-07 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770280+0000", 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770286+0000", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9190000000000004e-06 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770289+0000", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4759999999999998e-06 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.795326+0000", 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025036566 2024-09-16T13:29:32.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.795401+0000", 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5018999999999994e-05 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796418+0000", 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010167520000000001 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72245 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 262144~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388763660000002, 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026345113999999999, 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72245 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770293+0000", 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:32.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770295+0000", 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.64e-06 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770301+0000", 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4999999999999996e-06 2024-09-16T13:29:32.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770307+0000", 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3299999999999998e-06 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796445+0000", 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026138443000000001 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796541+0000", 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6248000000000004e-05 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796638+0000", 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6522999999999997e-05 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72246 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.396 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388589760000002, 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026527684999999999, 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72246 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770310+0000", 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0800000000000001e-07 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770312+0000", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.761e-06 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770319+0000", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5570000000000001e-06 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770322+0000", 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.366e-06 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796666+0000", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026343524 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796746+0000", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0755999999999997e-05 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796838+0000", 2024-09-16T13:29:32.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1413e-05 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72247 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388433599999995, 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026762220999999999, 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72247 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770326+0000", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770328+0000", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6759999999999999e-06 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770334+0000", 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9649999999999996e-06 2024-09-16T13:29:32.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770337+0000", 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796864+0000", 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026526554000000001 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.796943+0000", 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9411000000000003e-05 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797088+0000", 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144716 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72248 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:32.405 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388273169999991, 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026921207999999999, 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72248 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770342+0000", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770344+0000", 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.705e-06 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770349+0000", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770353+0000", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.535e-06 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797114+0000", 2024-09-16T13:29:32.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026761073999999999 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797186+0000", 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2657999999999995e-05 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797263+0000", 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6553999999999999e-05 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:72254 4.9 4:9138f27c:::rbd_data.11a911a8a02a.0000000000000010:1d [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.5388112320000005, 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02707977, 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 72254 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770358+0000", 2024-09-16T13:29:32.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770360+0000", 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6810000000000001e-06 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770365+0000", 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3900000000000001e-06 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.770369+0000", 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2679999999999999e-06 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797289+0000", 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026920342 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797359+0000", 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9739000000000004e-05 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.797438+0000", 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8798e-05 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.6670832650000005, 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:32.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:29:32.419 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 7.6499503710000001, 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:29:32.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:29:32.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.689561425, 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.424 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:29:32.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:29:32.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.673072704, 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:29:32.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.434 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.436 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:29:32.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:32.437 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:32.433+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:32.480 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-09-16T13:29:32.482 INFO:teuthology.orchestra.run.smithi076.stdout:4155 writesame 0x401bd4e thru 0x40202ca data_size 0x1(0x457d bytes) 2024-09-16T13:29:32.525 INFO:teuthology.orchestra.run.smithi076.stdout:4156 trunc from 0x89ae03a to 0x4afd27f 2024-09-16T13:29:32.536 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:32.534+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:4159 trunc from 0x4afd27f to 0x96499cc 2024-09-16T13:29:32.553 INFO:teuthology.orchestra.run.smithi076.stdout:4160 write 0x11a8f2c thru 0x11b542d (0xc502 bytes) 2024-09-16T13:29:32.560 INFO:teuthology.orchestra.run.smithi076.stdout:4161 punch from 0xf656a3 to 0xf6a366, (0x4cc3 bytes) 2024-09-16T13:29:32.560 INFO:teuthology.orchestra.run.smithi076.stdout:4162 writesame 0x85e9796 thru 0x85f8d2d data_size 0x74(0xf598 bytes) 2024-09-16T13:29:32.564 INFO:teuthology.orchestra.run.smithi076.stdout:4165 read 0x63bb320 thru 0x63c0f03 (0x5be4 bytes) 2024-09-16T13:29:32.564 INFO:teuthology.orchestra.run.smithi076.stdout:4166 write 0x5c04f0c thru 0x5c07281 (0x2376 bytes) 2024-09-16T13:29:32.566 INFO:teuthology.orchestra.run.smithi076.stdout:4167 write 0x4bcef7d thru 0x4bd0e61 (0x1ee5 bytes) 2024-09-16T13:29:32.567 INFO:teuthology.orchestra.run.smithi076.stdout:4168 read 0x1148677 thru 0x11541e4 (0xbb6e bytes) 2024-09-16T13:29:32.567 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:32.577 INFO:teuthology.orchestra.run.smithi076.stdout:4171 trunc from 0x96499cc to 0x1a0e91f 2024-09-16T13:29:32.577 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.577 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:32.577 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:32.578 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:32.614 INFO:teuthology.orchestra.run.smithi076.stdout:4172 read 0xddfda0 thru 0xdeb65b (0xb8bc bytes) 2024-09-16T13:29:32.629 INFO:teuthology.orchestra.run.smithi076.stdout:4173 writesame 0x9374124 thru 0x9378e9e data_size 0x1(0x4d7b bytes) 2024-09-16T13:29:32.630 INFO:teuthology.orchestra.run.smithi076.stdout:4174 clone 36 order 22 su 16384 sc 15 2024-09-16T13:29:32.637 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:32.635+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:32.689 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.689 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:32.689 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:32.689 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:32.689 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:32.741 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:32.739+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:32.797 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3216 4.15 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2890788~64977] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.155478318999997, 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021131289000000001, 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.798 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3216 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639486+0000", 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.639488+0000", 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642820+0000", 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033328479999999998 2024-09-16T13:29:32.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642841+0000", 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0282999999999999e-05 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642849+0000", 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.642902+0000", 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2772000000000002e-05 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643017+0000", 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115408 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643231+0000", 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000213291 2024-09-16T13:29:32.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.659344+0000", 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016112998999999999 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660548+0000", 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012041549999999999 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660583+0000", 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5503999999999998e-05 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660617+0000", 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4140999999999997e-05 2024-09-16T13:29:32.804 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.064947914000001, 2024-09-16T13:29:32.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:32.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:32.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.811 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:3537 4.15 e38/32 ondisk, result = 0)", 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.037818743999999, 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024289881999999999, 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757146+0000", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757148+0000", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757155+0000", 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7640000000000005e-06 2024-09-16T13:29:32.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757168+0000", 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2403e-05 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757173+0000", 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781292+0000", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024118876000000001 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781304+0000", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1554e-05 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781436+0000", 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013196900000000001 2024-09-16T13:29:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.622056592, 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:32.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6345 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'26, mlcod=40'21)", 2024-09-16T13:29:32.820 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.55172031, 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022935970999999999, 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243244+0000", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243246+0000", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.778e-06 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243260+0000", 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3900000000000001e-05 2024-09-16T13:29:32.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243283+0000", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2912e-05 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.243293+0000", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0787e-05 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257616+0000", 2024-09-16T13:29:32.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014322207 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.257640+0000", 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266159+0000", 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085193200000000004 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.266180+0000", 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0757000000000001e-05 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.108598563999998, 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.825 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:32.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.098509297, 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:32.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:27999 4.8 4:10b6e706:::rbd_data.11a9320547a2.0000000000000002:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:32.835 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.951720663000003, 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019308657999999999, 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 27999 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843244+0000", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843245+0000", 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4099999999999997e-07 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843249+0000", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2610000000000004e-06 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843260+0000", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0807e-05 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843265+0000", 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3210000000000004e-06 2024-09-16T13:29:32.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843297+0000", 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1597999999999998e-05 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.843347+0000", 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0866999999999999e-05 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.845836+0000", 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024888480000000001 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862422+0000", 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016585956999999998 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862476+0000", 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3263000000000001e-05 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862512+0000", 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6397999999999999e-05 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.862552+0000", 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0497000000000002e-05 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.841 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50162 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 581632~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546409921999999, 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019455525000000001, 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50162 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248554+0000", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248555+0000", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248557+0000", 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7850000000000001e-06 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248562+0000", 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8579999999999996e-06 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248566+0000", 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.207e-06 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267854+0000", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019288254000000001 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.267938+0000", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3678999999999996e-05 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268010+0000", 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2416000000000006e-05 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.845 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50163 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 598016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546395070999999, 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019564818000000001, 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50163 2024-09-16T13:29:32.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248569+0000", 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248570+0000", 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248577+0000", 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4320000000000002e-06 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248584+0000", 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3810000000000004e-06 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248587+0000", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3790000000000002e-06 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268040+0000", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019453427999999998 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268096+0000", 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5404e-05 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268134+0000", 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50164 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 614400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546373396, 2024-09-16T13:29:32.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019695615, 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50164 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248591+0000", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248593+0000", 2024-09-16T13:29:32.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9300000000000002e-06 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248599+0000", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7950000000000004e-06 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248603+0000", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268171+0000", 2024-09-16T13:29:32.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019568232000000001 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268243+0000", 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1872000000000006e-05 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268287+0000", 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3300000000000002e-05 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.854 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50165 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 630784~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546357208, 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019831332, 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.855 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50165 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248607+0000", 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248608+0000", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248610+0000", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.807e-06 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248619+0000", 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4779999999999997e-06 2024-09-16T13:29:32.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248623+0000", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0489999999999997e-06 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268313+0000", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019690151999999999 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268379+0000", 2024-09-16T13:29:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5792999999999998e-05 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268439+0000", 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9537000000000003e-05 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50166 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546337436999998, 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019843890999999999, 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50166 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248627+0000", 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.65e-07 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248629+0000", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.854e-06 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248635+0000", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248645+0000", 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7599999999999997e-06 2024-09-16T13:29:32.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268466+0000", 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019821442000000002 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268471+0000", 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.775e-06 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50167 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546315806999999, 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019857698999999999, 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50167 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248649+0000", 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.89e-07 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248651+0000", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.181e-06 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248657+0000", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0939999999999996e-06 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248661+0000", 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.54e-06 2024-09-16T13:29:32.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268499+0000", 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019837751000000001 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268506+0000", 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7440000000000007e-06 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50168 4.15 4.bb0416d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.546300093999999, 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019875693999999999, 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50168 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248664+0000", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248665+0000", 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:32.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248667+0000", 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.866e-06 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248672+0000", 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3410000000000002e-06 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.248675+0000", 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.264e-06 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268536+0000", 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019860932000000001 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.268540+0000", 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50929 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 327680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.498288109000001, 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019357040999999998, 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50929 2024-09-16T13:29:32.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296676+0000", 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296677+0000", 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9900000000000002e-07 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296678+0000", 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.401e-06 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296683+0000", 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-09-16T13:29:32.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296686+0000", 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8949999999999998e-06 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.315967+0000", 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019280701000000001 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316000+0000", 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3549e-05 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316033+0000", 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3272999999999998e-05 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:50930 4.15 4:ab6820dd:::rbd_data.11a9441c6517.0000000000000006:15 [read 344064~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:32.876 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.498275765999999, 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019423292000000002, 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50930 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296689+0000", 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5499999999999999e-07 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296690+0000", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.575e-06 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296696+0000", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3460000000000001e-06 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.296704+0000", 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6319999999999997e-06 2024-09-16T13:29:32.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316046+0000", 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019341649999999998 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316079+0000", 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2898999999999998e-05 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:05.316112+0000", 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2935000000000002e-05 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.819796270000001, 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.881 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:32.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:32.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.871637296999999, 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:29:32.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:29:32.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.893 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:58637 4.15 e74/70 ondisk, result = 0)", 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.862165468000001, 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026177753000000002, 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:32.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932799+0000", 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932800+0000", 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4309999999999999e-06 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:32.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932807+0000", 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1659999999999999e-06 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932826+0000", 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9256999999999999e-05 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.932835+0000", 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6239999999999997e-06 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958962+0000", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026126531000000001 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958971+0000", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5839999999999995e-06 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958977+0000", 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1600000000000003e-06 2024-09-16T13:29:32.897 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:32.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:32.898 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:32.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:32.900 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:32.897+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:33.000 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:32.998+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:33.039 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:33.039 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:33.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:33.132 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:33.130+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.187 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:33.187 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:33.187 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:33.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:33.237 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:33.235+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:33.303 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:33.301+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.647142176999999, 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.337 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:33.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:33.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.342 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.637570699000001, 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:33.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:33.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.790162017, 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.347 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:29:33.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:29:33.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.790151172000002, 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.352 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:33.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.790141757999997, 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:33.356 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043894280000004, 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.361 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:29:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043883586, 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:29:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:29:33.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043874598000002, 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.370 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:33.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043866856999998, 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.374 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:33.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:33.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043858298000004, 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.379 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:33.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.383 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043849420999997, 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:33.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043835555999998, 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:33.388 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:33.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043827088999997, 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:33.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:33.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.043818285999997, 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.398 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.420348228000002, 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.407119547000001, 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.893293358000001, 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:29:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:29:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.87623748, 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:29:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:29:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:29:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.9720869000000008, 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:29:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:29:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.9614478599999998, 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.436 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:33.437 INFO:teuthology.orchestra.run.smithi076.stdout:leaving image image_client.0-clone35 intact 2024-09-16T13:29:33.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:33.438 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:33.436+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:33.497 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-09-16T13:29:33.538 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:33.537+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:33.577 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.577 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:33.577 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:33.577 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:33.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:33.639 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:33.638+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:33.651 INFO:teuthology.orchestra.run.smithi076.stdout:4176 read 0x59bfceb thru 0x59c77c6 (0x7adc bytes) 2024-09-16T13:29:33.652 INFO:teuthology.orchestra.run.smithi076.stdout:4177 write 0x6d3097e thru 0x6d3fe0b (0xf48e bytes) 2024-09-16T13:29:33.670 INFO:teuthology.orchestra.run.smithi076.stdout:4178 write 0x534f2e3 thru 0x5354f8d (0x5cab bytes) 2024-09-16T13:29:33.681 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.681 INFO:teuthology.orchestra.run.smithi076.stdout:4180 write 0x7ac0453 thru 0x7ac5b2e (0x56dc bytes) 2024-09-16T13:29:33.687 INFO:teuthology.orchestra.run.smithi076.stdout:4181 read 0x6ec53e0 thru 0x6ecf71a (0xa33b bytes) 2024-09-16T13:29:33.687 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.687 INFO:teuthology.orchestra.run.smithi076.stdout:4184 write 0x8d6c3ba thru 0x8d78ce0 (0xc927 bytes) 2024-09-16T13:29:33.694 INFO:teuthology.orchestra.run.smithi076.stdout:4186 punch from 0x7311883 to 0x731aa4d, (0x91ca bytes) 2024-09-16T13:29:33.700 INFO:teuthology.orchestra.run.smithi076.stdout:4187 trunc from 0x9378e9f to 0x4c2bf27 2024-09-16T13:29:33.731 INFO:teuthology.orchestra.run.smithi076.stdout:4189 punch from 0x1e72d0c to 0x1e826b9, (0xf9ad bytes) 2024-09-16T13:29:33.740 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:33.739+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:33.741 INFO:teuthology.orchestra.run.smithi076.stdout:4190 write 0x8e46a90 thru 0x8e4fc4b (0x91bc bytes) 2024-09-16T13:29:33.742 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:33.743 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:33.746 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.746 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.746 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.749 INFO:teuthology.orchestra.run.smithi076.stdout:4195 trunc from 0x8e4fc4c to 0x5f1bace 2024-09-16T13:29:33.752 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:33.768 INFO:teuthology.orchestra.run.smithi076.stdout:4196 trunc from 0x5f1bace to 0x83334a5 2024-09-16T13:29:33.773 INFO:teuthology.orchestra.run.smithi076.stdout:4197 writesame 0x4edf182 thru 0x4ee9e38 data_size 0x25(0xacb7 bytes) 2024-09-16T13:29:33.775 INFO:teuthology.orchestra.run.smithi076.stdout:4198 punch from 0x30be307 to 0x30cbd6d, (0xda66 bytes) 2024-09-16T13:29:33.787 INFO:teuthology.orchestra.run.smithi076.stdout:4199 writesame 0xdf10b38 thru 0xdf148e7 data_size 0x38(0x3db0 bytes) 2024-09-16T13:29:33.787 INFO:teuthology.orchestra.run.smithi076.stdout:4200 writesame 0x902ff98 thru 0x903e288 data_size 0x29(0xe2f1 bytes) 2024-09-16T13:29:33.798 INFO:teuthology.orchestra.run.smithi076.stdout:4201 writesame 0x51d0b8b thru 0x51ddd21 data_size 0x93(0xd197 bytes) 2024-09-16T13:29:33.800 INFO:teuthology.orchestra.run.smithi076.stdout:4202 read 0xa763e9b thru 0xa76555f (0x16c5 bytes) 2024-09-16T13:29:33.800 INFO:teuthology.orchestra.run.smithi076.stdout:4203 read 0x9bbf572 thru 0x9bc07ae (0x123d bytes) 2024-09-16T13:29:33.800 INFO:teuthology.orchestra.run.smithi076.stdout:4204 writesame 0x7b1610c thru 0x7b21413 data_size 0x44(0xb308 bytes) 2024-09-16T13:29:33.802 INFO:teuthology.orchestra.run.smithi076.stdout:4205 write 0x761da53 thru 0x7622459 (0x4a07 bytes) 2024-09-16T13:29:33.804 INFO:teuthology.orchestra.run.smithi076.stdout:4208 punch from 0x88722e6 to 0x8874d02, (0x2a1c bytes) 2024-09-16T13:29:33.804 INFO:teuthology.orchestra.run.smithi076.stdout:4211 read 0xdbf7f66 thru 0xdc05b52 (0xdbed bytes) 2024-09-16T13:29:33.810 INFO:teuthology.orchestra.run.smithi076.stdout:4212 writesame 0xea75ad1 thru 0xea772ac data_size 0x6(0x17dc bytes) 2024-09-16T13:29:33.810 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:33.810 INFO:teuthology.orchestra.run.smithi076.stdout:4214 writesame 0x44a5a7c thru 0x44b251a data_size 0x1(0xca9f bytes) 2024-09-16T13:29:33.841 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:33.839+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:33.850 INFO:teuthology.orchestra.run.smithi076.stdout:4215 read 0x3f4e27f thru 0x3f4f127 (0xea9 bytes) 2024-09-16T13:29:33.850 INFO:teuthology.orchestra.run.smithi076.stdout:4217 punch from 0xb679240 to 0xb684fc3, (0xbd83 bytes) 2024-09-16T13:29:33.850 INFO:teuthology.orchestra.run.smithi076.stdout:4218 read 0x56a6232 thru 0x56af5d2 (0x93a1 bytes) 2024-09-16T13:29:33.851 INFO:teuthology.orchestra.run.smithi076.stdout:4219 read 0x9d75944 thru 0x9d80ce9 (0xb3a6 bytes) 2024-09-16T13:29:33.851 INFO:teuthology.orchestra.run.smithi076.stdout:4221 write 0x7e2b858 thru 0x7e2ddb3 (0x255c bytes) 2024-09-16T13:29:33.852 INFO:teuthology.orchestra.run.smithi076.stdout:4222 writesame 0x1b4d2b1 thru 0x1b59168 data_size 0x44(0xbeb8 bytes) 2024-09-16T13:29:33.901 INFO:teuthology.orchestra.run.smithi076.stdout:4223 write 0x63f6072 thru 0x64015b1 (0xb540 bytes) 2024-09-16T13:29:33.903 INFO:teuthology.orchestra.run.smithi076.stdout:4225 read 0x2ef665f thru 0x2f0041d (0x9dbf bytes) 2024-09-16T13:29:33.903 INFO:teuthology.orchestra.run.smithi076.stdout:4226 writesame 0xa597942 thru 0xa59f836 data_size 0x1(0x7ef5 bytes) 2024-09-16T13:29:33.911 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:33.911 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:33.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 88.619406667000007, 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:33.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:42 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917934074000001, 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0076953830000000001, 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990673+0000", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990674+0000", 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6199999999999998e-07 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990676+0000", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.311e-06 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990680+0000", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990682+0000", 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:29:33.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992638+0000", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001956012 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992719+0000", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1137999999999995e-05 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.992848+0000", 2024-09-16T13:29:33.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012877900000000001 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996364+0000", 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035157859999999999 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998289+0000", 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019255839999999999 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998333+0000", 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4351999999999998e-05 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998369+0000", 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5123000000000002e-05 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:33.924 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917758871999993, 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:33.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.930 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917596259000007, 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:33.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:33.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.936 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917575212000003, 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.942 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917487731999998, 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.943 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:33.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917465222999994, 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.949 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917418896000001, 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:33.960 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917252138999999, 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.961 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:33.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:33.966 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.917237373999995, 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.967 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:33.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:33.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:33.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.571915056999998, 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.973 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:33.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:33.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.571877379, 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:33.979 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:33.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.571854565999999, 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:33.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.571769352999993, 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:33.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:33.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:33.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.265190427999997, 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:33.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.239282193999998, 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:34.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:34.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:34.005 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.170517039000003, 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:34.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.668974221999999, 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:34.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:34.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.016 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.474048903, 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:34.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:29:34.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.3181718680000001, 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:29:34.021 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:29:34.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:29:34.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:34.026 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:34.026 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:34.028 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:34.025+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:34.028 INFO:teuthology.orchestra.run.smithi076.stdout:4228 writesame 0x2daf238 thru 0x2db2138 data_size 0x15(0x2f01 bytes) 2024-09-16T13:29:34.028 INFO:teuthology.orchestra.run.smithi076.stdout:4229 punch from 0x2adbf74 to 0x2ae0e9e, (0x4f2a bytes) 2024-09-16T13:29:34.028 INFO:teuthology.orchestra.run.smithi076.stdout:4230 write 0x59f5360 thru 0x59fcf9b (0x7c3c bytes) 2024-09-16T13:29:34.028 INFO:teuthology.orchestra.run.smithi076.stdout:4231 read 0x384d254 thru 0x384e6b2 (0x145f bytes) 2024-09-16T13:29:34.029 INFO:teuthology.orchestra.run.smithi076.stdout:4232 read 0x2fb587e thru 0x2fc434d (0xead0 bytes) 2024-09-16T13:29:34.029 INFO:teuthology.orchestra.run.smithi076.stdout:4234 read 0x92ed0ec thru 0x92f181a (0x472f bytes) 2024-09-16T13:29:34.029 INFO:teuthology.orchestra.run.smithi076.stdout:4235 trunc from 0xea772ad to 0x2ea333f 2024-09-16T13:29:34.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:34.051 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:34.129 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:34.127+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:4236 trunc from 0x2ea333f to 0xca8d78d 2024-09-16T13:29:34.134 INFO:teuthology.orchestra.run.smithi076.stdout:4237 write 0xb6812c3 thru 0xb6832bc (0x1ffa bytes) 2024-09-16T13:29:34.137 INFO:teuthology.orchestra.run.smithi076.stdout:4238 write 0x96b51fc thru 0x96bf466 (0xa26b bytes) 2024-09-16T13:29:34.142 INFO:teuthology.orchestra.run.smithi076.stdout:4239 write 0x5db0859 thru 0x5dba900 (0xa0a8 bytes) 2024-09-16T13:29:34.146 INFO:teuthology.orchestra.run.smithi076.stdout:4243 write 0x8e63b93 thru 0x8e703f9 (0xc867 bytes) 2024-09-16T13:29:34.151 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:34.151 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:34.151 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:34.151 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:34.156 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.156 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.156 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.156 INFO:teuthology.orchestra.run.smithi076.stdout:4250 read 0x2c45af8 thru 0x2c4d1e7 (0x76f0 bytes) 2024-09-16T13:29:34.156 INFO:teuthology.orchestra.run.smithi076.stdout:4251 read 0xf21f12 thru 0xf2a849 (0x8938 bytes) 2024-09-16T13:29:34.157 INFO:teuthology.orchestra.run.smithi076.stdout:4252 read 0x1506f98 thru 0x1512900 (0xb969 bytes) 2024-09-16T13:29:34.158 INFO:teuthology.orchestra.run.smithi076.stdout:4253 punch from 0x4c225dd to 0x4c29c6e, (0x7691 bytes) 2024-09-16T13:29:34.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:34.163 INFO:teuthology.orchestra.run.smithi076.stdout:4254 trunc from 0xca8d78d to 0x8d62928 2024-09-16T13:29:34.185 INFO:teuthology.orchestra.run.smithi076.stdout:4255 write 0xb6e3555 thru 0xb6e3792 (0x23e bytes) 2024-09-16T13:29:34.187 INFO:teuthology.orchestra.run.smithi076.stdout:4256 write 0xaaa0e64 thru 0xaaaaf27 (0xa0c4 bytes) 2024-09-16T13:29:34.190 INFO:teuthology.orchestra.run.smithi076.stdout:4258 write 0x8e6d3c8 thru 0x8e76f8b (0x9bc4 bytes) 2024-09-16T13:29:34.196 INFO:teuthology.orchestra.run.smithi076.stdout:4259 clone 37 order 19 su 16384 sc 14 2024-09-16T13:29:34.209 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:34.224 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:34.224 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:34.229 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:34.228+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:34.239 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:34.239 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:34.255 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:34.267 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.267 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.270 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:34.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:34.302 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:34.318 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:34.318 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:34.330 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:34.329+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:34.391 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.391 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:34.405 INFO:teuthology.orchestra.run.smithi076.stdout:flattening image image_client.0-clone36 2024-09-16T13:29:34.431 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:34.429+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:34.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:29:34.504 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.504 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:34.532 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:34.530+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:34.545 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-09-16T13:29:34.625 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.625 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:34.634 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:34.632+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:34.734 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:34.732+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:34.748 INFO:teuthology.orchestra.run.smithi076.stdout:4260 write 0xea31693 thru 0xea335c8 (0x1f36 bytes) 2024-09-16T13:29:34.751 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.751 INFO:teuthology.orchestra.run.smithi076.stdout:4264 read 0xcccf198 thru 0xccd2160 (0x2fc9 bytes) 2024-09-16T13:29:34.752 INFO:teuthology.orchestra.run.smithi076.stdout:4265 punch from 0x915ec6 to 0x91d674, (0x77ae bytes) 2024-09-16T13:29:34.752 INFO:teuthology.orchestra.run.smithi076.stdout:4266 punch from 0x672d72d to 0x6730258, (0x2b2b bytes) 2024-09-16T13:29:34.763 INFO:teuthology.orchestra.run.smithi076.stdout:4267 trunc from 0xea335c9 to 0x4cdba5e 2024-09-16T13:29:34.784 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.784 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:34.801 INFO:teuthology.orchestra.run.smithi076.stdout:4268 writesame 0x91fc7f5 thru 0x920891d data_size 0x3(0xc129 bytes) 2024-09-16T13:29:34.801 INFO:teuthology.orchestra.run.smithi076.stdout:4269 read 0x858b98a thru 0x858f117 (0x378e bytes) 2024-09-16T13:29:34.801 INFO:teuthology.orchestra.run.smithi076.stdout:4270 read 0x3a42226 thru 0x3a4a9f7 (0x87d2 bytes) 2024-09-16T13:29:34.812 INFO:teuthology.orchestra.run.smithi076.stdout:4271 writesame 0xcc54d24 thru 0xcc60daa data_size 0x7(0xc087 bytes) 2024-09-16T13:29:34.812 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.812 INFO:teuthology.orchestra.run.smithi076.stdout:4273 punch from 0x749f1f4 to 0x74acb04, (0xd910 bytes) 2024-09-16T13:29:34.812 INFO:teuthology.orchestra.run.smithi076.stdout:4274 read 0x4f9a744 thru 0x4fa5036 (0xa8f3 bytes) 2024-09-16T13:29:34.812 INFO:teuthology.orchestra.run.smithi076.stdout:4275 write 0x684c4ca thru 0x684fee2 (0x3a19 bytes) 2024-09-16T13:29:34.817 INFO:teuthology.orchestra.run.smithi076.stdout:4276 trunc from 0xcc60dab to 0xd5ec561 2024-09-16T13:29:34.822 INFO:teuthology.orchestra.run.smithi076.stdout:4277 writesame 0x69575ee thru 0x6963e8b data_size 0x1(0xc89e bytes) 2024-09-16T13:29:34.835 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:34.834+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:34.860 INFO:teuthology.orchestra.run.smithi076.stdout:4278 read 0xc0af6dc thru 0xc0b428c (0x4bb1 bytes) 2024-09-16T13:29:34.860 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:34.860 INFO:teuthology.orchestra.run.smithi076.stdout:4280 trunc from 0xd5ec561 to 0xd2c7e89 2024-09-16T13:29:34.880 INFO:teuthology.orchestra.run.smithi076.stdout:4281 trunc from 0xd2c7e89 to 0x2b43bd9 2024-09-16T13:29:34.923 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:34.923 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:34.937 INFO:teuthology.orchestra.run.smithi076.stdout:4282 trunc from 0x2b43bd9 to 0xb6003f6 2024-09-16T13:29:34.937 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:34.935+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:34.941 INFO:teuthology.orchestra.run.smithi076.stdout:4283 writesame 0x8d4842b thru 0x8d4ba0b data_size 0x1(0x35e1 bytes) 2024-09-16T13:29:34.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:34.967 INFO:teuthology.orchestra.run.smithi076.stdout:4284 write 0xca1524d thru 0xca221f9 (0xcfad bytes) 2024-09-16T13:29:34.973 INFO:teuthology.orchestra.run.smithi076.stdout:4287 read 0x992c6c1 thru 0x992d758 (0x1098 bytes) 2024-09-16T13:29:34.973 INFO:teuthology.orchestra.run.smithi076.stdout:4288 writesame 0x807d5bc thru 0x808603d data_size 0x1(0x8a82 bytes) 2024-09-16T13:29:35.005 INFO:teuthology.orchestra.run.smithi076.stdout:4290 trunc from 0xca221fa to 0x9f2561a 2024-09-16T13:29:35.015 INFO:teuthology.orchestra.run.smithi076.stdout:4292 read 0x5f0df2 thru 0x5fdfe8 (0xd1f7 bytes) 2024-09-16T13:29:35.017 INFO:teuthology.orchestra.run.smithi076.stdout:4293 trunc from 0x9f2561a to 0xb4454d4 2024-09-16T13:29:35.022 INFO:teuthology.orchestra.run.smithi076.stdout:4296 punch from 0x10131a9 to 0x101674c, (0x35a3 bytes) 2024-09-16T13:29:35.022 INFO:teuthology.orchestra.run.smithi076.stdout:4297 writesame 0x46e590c thru 0x46f07b7 data_size 0x1(0xaeac bytes) 2024-09-16T13:29:35.037 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:35.036+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:35.059 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.059 INFO:teuthology.orchestra.run.smithi076.stdout:4300 read 0x94d8b9e thru 0x94e2e53 (0xa2b6 bytes) 2024-09-16T13:29:35.059 INFO:teuthology.orchestra.run.smithi076.stdout:4301 read 0x31b7d5f thru 0x31c43ea (0xc68c bytes) 2024-09-16T13:29:35.059 INFO:teuthology.orchestra.run.smithi076.stdout:4302 write 0x8271477 thru 0x827bb52 (0xa6dc bytes) 2024-09-16T13:29:35.063 INFO:teuthology.orchestra.run.smithi076.stdout:4305 trunc from 0xb4454d4 to 0xa67d83a 2024-09-16T13:29:35.069 INFO:teuthology.orchestra.run.smithi076.stdout:4306 read 0x7b3bea6 thru 0x7b42831 (0x698c bytes) 2024-09-16T13:29:35.069 INFO:teuthology.orchestra.run.smithi076.stdout:4308 read 0x2fcf44b thru 0x2fde89c (0xf452 bytes) 2024-09-16T13:29:35.069 INFO:teuthology.orchestra.run.smithi076.stdout:4309 punch from 0x6988090 to 0x698b598, (0x3508 bytes) 2024-09-16T13:29:35.071 INFO:teuthology.orchestra.run.smithi076.stdout:4310 trunc from 0xa67d83a to 0xb6c8acb 2024-09-16T13:29:35.073 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.073 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.076 INFO:teuthology.orchestra.run.smithi076.stdout:4311 writesame 0x1ce9a9b thru 0x1cea34a data_size 0x8(0x8b0 bytes) 2024-09-16T13:29:35.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:35.093 INFO:teuthology.orchestra.run.smithi076.stdout:4312 trunc from 0xb6c8acb to 0x18826b2 2024-09-16T13:29:35.138 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:35.136+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:35.146 INFO:teuthology.orchestra.run.smithi076.stdout:4313 write 0x1db3970 thru 0x1dbdf7f (0xa610 bytes) 2024-09-16T13:29:35.165 INFO:teuthology.orchestra.run.smithi076.stdout:4315 trunc from 0x1dbdf80 to 0xd46b7b5 2024-09-16T13:29:35.170 INFO:teuthology.orchestra.run.smithi076.stdout:4316 punch from 0x2218ab to 0x22d93a, (0xc08f bytes) 2024-09-16T13:29:35.170 INFO:teuthology.orchestra.run.smithi076.stdout:4318 writesame 0x23e69ab thru 0x23eb9e8 data_size 0x1(0x503e bytes) 2024-09-16T13:29:35.184 INFO:teuthology.orchestra.run.smithi076.stdout:4319 write 0x9becd60 thru 0x9beddf3 (0x1094 bytes) 2024-09-16T13:29:35.187 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.187 INFO:teuthology.orchestra.run.smithi076.stdout:4321 read 0x26da8a8 thru 0x26e3a54 (0x91ad bytes) 2024-09-16T13:29:35.193 INFO:teuthology.orchestra.run.smithi076.stdout:4322 write 0xd5c2cd3 thru 0xd5ca25e (0x758c bytes) 2024-09-16T13:29:35.198 INFO:teuthology.orchestra.run.smithi076.stdout:4323 trunc from 0xd5ca25f to 0xe5d69e4 2024-09-16T13:29:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:4324 read 0x462dd38 thru 0x4630213 (0x24dc bytes) 2024-09-16T13:29:35.203 INFO:teuthology.orchestra.run.smithi076.stdout:4326 write 0x982929d thru 0x982a30d (0x1071 bytes) 2024-09-16T13:29:35.206 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.206 INFO:teuthology.orchestra.run.smithi076.stdout:4328 punch from 0x93f20f5 to 0x93fe2fe, (0xc209 bytes) 2024-09-16T13:29:35.206 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.206 INFO:teuthology.orchestra.run.smithi076.stdout:4330 writesame 0xa6ae5d8 thru 0xa6b1c54 data_size 0xd(0x367d bytes) 2024-09-16T13:29:35.209 INFO:teuthology.orchestra.run.smithi076.stdout:4331 punch from 0x170c6a2 to 0x1711b36, (0x5494 bytes) 2024-09-16T13:29:35.210 INFO:teuthology.orchestra.run.smithi076.stdout:4332 writesame 0xf394be thru 0xf476ba data_size 0x1(0xe1fd bytes) 2024-09-16T13:29:35.213 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.213 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.224 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:35.250 INFO:teuthology.orchestra.run.smithi076.stdout:4333 punch from 0x1b8adeb to 0x1b8e16a, (0x337f bytes) 2024-09-16T13:29:35.251 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:35.248+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:35.257 INFO:teuthology.orchestra.run.smithi076.stdout:4334 trunc from 0xe5d69e4 to 0x81620e7 2024-09-16T13:29:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:4335 read 0x68348fc thru 0x683eeb3 (0xa5b8 bytes) 2024-09-16T13:29:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:4337 read 0x77252d5 thru 0x7728aab (0x37d7 bytes) 2024-09-16T13:29:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.274 INFO:teuthology.orchestra.run.smithi076.stdout:4339 trunc from 0x81620e7 to 0xe377afe 2024-09-16T13:29:35.279 INFO:teuthology.orchestra.run.smithi076.stdout:4340 write 0x82a92d3 thru 0x82a98f8 (0x626 bytes) 2024-09-16T13:29:35.286 INFO:teuthology.orchestra.run.smithi076.stdout:4341 write 0xe37a6f5 thru 0xe37ae8d (0x799 bytes) 2024-09-16T13:29:35.293 INFO:teuthology.orchestra.run.smithi076.stdout:4342 trunc from 0xe37ae8e to 0xbf5d75f 2024-09-16T13:29:35.303 INFO:teuthology.orchestra.run.smithi076.stdout:4343 writesame 0x1475d40 thru 0x14773ef data_size 0x30(0x16b0 bytes) 2024-09-16T13:29:35.308 INFO:teuthology.orchestra.run.smithi076.stdout:4344 read 0x23bbf29 thru 0x23bf8ed (0x39c5 bytes) 2024-09-16T13:29:35.308 INFO:teuthology.orchestra.run.smithi076.stdout:4345 write 0x91af71d thru 0x91bcd1f (0xd603 bytes) 2024-09-16T13:29:35.317 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.323 INFO:teuthology.orchestra.run.smithi076.stdout:4347 trunc from 0xbf5d75f to 0x6ac407f 2024-09-16T13:29:35.336 INFO:teuthology.orchestra.run.smithi076.stdout:4348 read 0x232f5ca thru 0x2331b6e (0x25a5 bytes) 2024-09-16T13:29:35.339 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:35.337+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:35.345 INFO:teuthology.orchestra.run.smithi076.stdout:4349 write 0x9b4a959 thru 0x9b4f643 (0x4ceb bytes) 2024-09-16T13:29:35.350 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:29:35.353 INFO:teuthology.orchestra.run.smithi076.stdout:4350 read 0x5e8bf6f thru 0x5e96fa7 (0xb039 bytes) 2024-09-16T13:29:35.353 INFO:teuthology.orchestra.run.smithi076.stdout:4351 flatten 2024-09-16T13:29:35.370 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.370 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:35.451 INFO:teuthology.orchestra.run.smithi076.stdout:4352 writesame 0xaf8b54e thru 0xaf9a31e data_size 0x1(0xedd1 bytes) 2024-09-16T13:29:35.451 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.451 INFO:teuthology.orchestra.run.smithi076.stdout:4354 read 0x5cceac6 thru 0x5cd1578 (0x2ab3 bytes) 2024-09-16T13:29:35.451 INFO:teuthology.orchestra.run.smithi076.stdout:4355 punch from 0x6bb7231 to 0x6bc25c3, (0xb392 bytes) 2024-09-16T13:29:35.451 INFO:teuthology.orchestra.run.smithi076.stdout:4356 read 0x6c56dce thru 0x6c65d1c (0xef4f bytes) 2024-09-16T13:29:35.451 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:35.449+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:35.461 INFO:teuthology.orchestra.run.smithi076.stdout:4358 trunc from 0xaf9a31f to 0x2820799 2024-09-16T13:29:35.488 INFO:teuthology.orchestra.run.smithi076.stdout:4360 writesame 0xb7bd224 thru 0xb7c1301 data_size 0x1(0x40de bytes) 2024-09-16T13:29:35.496 INFO:teuthology.orchestra.run.smithi076.stdout:4361 trunc from 0xb7c1302 to 0x53d939b 2024-09-16T13:29:35.501 INFO:teuthology.orchestra.run.smithi076.stdout:4363 read 0x3a87577 thru 0x3a92bbb (0xb645 bytes) 2024-09-16T13:29:35.502 INFO:teuthology.orchestra.run.smithi076.stdout:4364 read 0x792a66 thru 0x792ff3 (0x58e bytes) 2024-09-16T13:29:35.502 INFO:teuthology.orchestra.run.smithi076.stdout:4365 read 0x3e6f24b thru 0x3e7f024 (0xfdda bytes) 2024-09-16T13:29:35.517 INFO:teuthology.orchestra.run.smithi076.stdout:4366 writesame 0xceb5eda thru 0xcebf944 data_size 0x1(0x9a6b bytes) 2024-09-16T13:29:35.517 INFO:teuthology.orchestra.run.smithi076.stdout:4368 write 0x6e464ff thru 0x6e495a2 (0x30a4 bytes) 2024-09-16T13:29:35.521 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:35.521 INFO:teuthology.orchestra.run.smithi076.stdout:4370 write 0xa0b1c6b thru 0xa0b3595 (0x192b bytes) 2024-09-16T13:29:35.524 INFO:teuthology.orchestra.run.smithi076.stdout:4371 read 0x304903d thru 0x304bd93 (0x2d57 bytes) 2024-09-16T13:29:35.524 INFO:teuthology.orchestra.run.smithi076.stdout:4372 read 0x1f85d7b thru 0x1f87c99 (0x1f1f bytes) 2024-09-16T13:29:35.524 INFO:teuthology.orchestra.run.smithi076.stdout:4373 write 0x75b0a0 thru 0x7674a4 (0xc405 bytes) 2024-09-16T13:29:35.527 INFO:teuthology.orchestra.run.smithi076.stdout:4374 write 0x6df13d thru 0x6e67d8 (0x769c bytes) 2024-09-16T13:29:35.536 INFO:teuthology.orchestra.run.smithi076.stdout:4375 writesame 0xedf55b9 thru 0xee0469e data_size 0xe(0xf0e6 bytes) 2024-09-16T13:29:35.536 INFO:teuthology.orchestra.run.smithi076.stdout:4376 writesame 0x9c1b566 thru 0x9c23ddd data_size 0xb(0x8878 bytes) 2024-09-16T13:29:35.537 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.538 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.540 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:35.538+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:35.543 INFO:teuthology.orchestra.run.smithi076.stdout:4377 read 0xac2886 thru 0xac747f (0x4bfa bytes) 2024-09-16T13:29:35.543 INFO:teuthology.orchestra.run.smithi076.stdout:4378 write 0xccf0bad thru 0xccf2460 (0x18b4 bytes) 2024-09-16T13:29:35.545 INFO:teuthology.orchestra.run.smithi076.stdout:4379 write 0xe2444ec thru 0xe250c59 (0xc76e bytes) 2024-09-16T13:29:35.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:35.555 INFO:teuthology.orchestra.run.smithi076.stdout:4380 read 0xb7ac23 thru 0xb8123c (0x661a bytes) 2024-09-16T13:29:35.555 INFO:teuthology.orchestra.run.smithi076.stdout:4381 write 0x642d485 thru 0x6430684 (0x3200 bytes) 2024-09-16T13:29:35.573 INFO:teuthology.orchestra.run.smithi076.stdout:4382 trunc from 0xee0469f to 0x4985a00 2024-09-16T13:29:35.586 INFO:teuthology.orchestra.run.smithi076.stdout:4383 write 0x2e340aa thru 0x2e34317 (0x26e bytes) 2024-09-16T13:29:35.603 INFO:teuthology.orchestra.run.smithi076.stdout:4385 writesame 0xb34a5b5 thru 0xb34a668 data_size 0x5(0xb4 bytes) 2024-09-16T13:29:35.603 INFO:teuthology.orchestra.run.smithi076.stdout:4386 write 0x88f225b thru 0x88f48c0 (0x2666 bytes) 2024-09-16T13:29:35.609 INFO:teuthology.orchestra.run.smithi076.stdout:4387 write 0x7796222 thru 0x779b40d (0x51ec bytes) 2024-09-16T13:29:35.614 INFO:teuthology.orchestra.run.smithi076.stdout:4389 write 0x3ebcfa3 thru 0x3ecb184 (0xe1e2 bytes) 2024-09-16T13:29:35.638 INFO:teuthology.orchestra.run.smithi076.stdout:4390 trunc from 0xb34a669 to 0x19cb288 2024-09-16T13:29:35.640 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:35.638+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:35.662 INFO:teuthology.orchestra.run.smithi076.stdout:4391 read 0x262b7d thru 0x2659e0 (0x2e64 bytes) 2024-09-16T13:29:35.674 INFO:teuthology.orchestra.run.smithi076.stdout:4393 write 0x63d1ec9 thru 0x63dc212 (0xa34a bytes) 2024-09-16T13:29:35.676 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.676 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.681 INFO:teuthology.orchestra.run.smithi076.stdout:4394 read 0x2cd28f6 thru 0x2cde232 (0xb93d bytes) 2024-09-16T13:29:35.681 INFO:teuthology.orchestra.run.smithi076.stdout:4395 read 0x17580f6 thru 0x175e20f (0x611a bytes) 2024-09-16T13:29:35.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:35.694 INFO:teuthology.orchestra.run.smithi076.stdout:4396 write 0xc9d91c2 thru 0xc9dd4dc (0x431b bytes) 2024-09-16T13:29:35.705 INFO:teuthology.orchestra.run.smithi076.stdout:4398 write 0xecf2f15 thru 0xed00bcf (0xdcbb bytes) 2024-09-16T13:29:35.716 INFO:teuthology.orchestra.run.smithi076.stdout:4400 clone 38 order 19 su 32768 sc 9 2024-09-16T13:29:35.741 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:35.739+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:35.788 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.788 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:35.842 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:35.840+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:35.935 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:35.935 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:35.942 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:35.941+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:35.943 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:36.044 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:36.042+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:36.082 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:36.082 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:36.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:36.145 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:36.143+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:36.189 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:36.189 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:36.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:36.246 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:36.245+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:36.296 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:36.296 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:36.304 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:36.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:36.335 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:36.336 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:36.336 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T13:29:36.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:29:36.373 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:36.371+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:36.408 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:36.409 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:36.418 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:36.462 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:36.460+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:36.477 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:36.477 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:36.496 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:36.496 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:36.513 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:36.513 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:36.516 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:36.516 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:36.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:36.529 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:36.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:36.546 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:36.562 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:36.566 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:36.564+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:36.580 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:36.580 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:36.582 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-09-16T13:29:36.596 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:36.596 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:36.650 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:36.649+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:36.664 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:36.664 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:36.664 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:36.665 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:36.665 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:36.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:36.682 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 91, 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:29:36.345578+0000", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:29:06.165279+0000", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:29:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 4, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:29:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:29:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:29:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "pool_client.0", 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:25.583589+0000", 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "91", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 41, 2024-09-16T13:29:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 91, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:36.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:36.706 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:29:36.707 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 69, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.710 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:29:36.712 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.713 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 56, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.714 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.715 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.716 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:29:36.718 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.719 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.6766510009765625, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 57, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:29:36.721 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.722 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.723 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:29:36.724 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.725 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:29:36.726 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.727 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:29:36.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:29:36.729 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.731 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.732 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.733 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.735 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.736 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:29:36.738 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.741 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:29:36.742 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:29:36.746 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.747 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:36.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:36.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.751 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:36.753 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.754 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.756 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.757 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.758 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.5", 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:36.759 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.2", 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:29:36.761 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [], 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:29:36.762 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:36.763 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:36.765 INFO:tasks.thrashosds.thrasher:j is {'epoch': 91, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:29:36.345578+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:29:06.165279+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:28:25.583589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 41, 'snap_epoch': 91, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6766510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 57, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [6, 1]}], 'pg_upmap_items': [{'pgid': '2.2', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:29:36.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:29:36.766 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:36.763+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:36.773 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.083561693, 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:36.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:36.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.073990215000002, 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:36.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.783 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7496 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 12779520~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.226581533000001, 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037366721999999998, 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7496 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.784 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.543999+0000", 2024-09-16T13:29:36.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544000+0000", 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544004+0000", 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.631e-06 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544008+0000", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2019999999999997e-06 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581008+0000", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037000330999999997 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581060+0000", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1239000000000002e-05 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581366+0000", 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030629700000000003 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7497 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 13697024~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.226570688000002, 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037671028000000002, 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.788 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7497 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544010+0000", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544011+0000", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999997e-07 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544014+0000", 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.807e-06 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544016+0000", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581385+0000", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037369312000000002 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581442+0000", 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6598000000000003e-05 2024-09-16T13:29:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581681+0000", 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023912200000000001 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.226561273999998, 2024-09-16T13:29:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:36.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:36.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8156 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7208960~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480313795999997, 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037113670000000001, 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8156 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290267+0000", 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:36.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290268+0000", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8899999999999998e-07 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290271+0000", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.841e-06 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290273+0000", 2024-09-16T13:29:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.798e-06 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327095+0000", 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036822000000000001 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327138+0000", 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3643999999999998e-05 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327381+0000", 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024229500000000001 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8165 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 851968~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480303102000001, 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037560205999999999, 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.801 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8165 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290278+0000", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290279+0000", 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0610000000000001e-06 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290283+0000", 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0450000000000002e-06 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290285+0000", 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.869e-06 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327416+0000", 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037131130999999998 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327506+0000", 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9677000000000002e-05 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327838+0000", 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033221299999999999 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.805 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480294114000003, 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:36.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:36.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:36.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480286372999998, 2024-09-16T13:29:36.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:36.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:36.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480277813999997, 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480268936999998, 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:36.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.823 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480255071999999, 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480246604999998, 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:36.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:36.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.480237801999998, 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.835 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.856767744000003, 2024-09-16T13:29:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:36.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:36.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.843539063000001, 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:36.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:36.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.329712874000002, 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:29:36.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:36.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.312656996000001, 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:29:36.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:29:36.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.408506416, 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:29:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:29:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:29:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.397867376000001, 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:29:36.864 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:29:36.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:36.868 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:36.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:36.871 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:36.867+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4402 trunc from 0xed00bd0 to 0x6234f03 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4404 punch from 0xc69b6f to 0xc6c733, (0x2bc4 bytes) 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4405 trunc from 0x6234f03 to 0xdbc0b48 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4406 trunc from 0xdbc0b48 to 0x6fe40e0 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4407 read 0x34d4f71 thru 0x34ddf39 (0x8fc9 bytes) 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4408 read 0x5e567f1 thru 0x5e644ef (0xdcff bytes) 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4410 punch from 0x1d6ba19 to 0x1d75c16, (0xa1fd bytes) 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4412 writesame 0xc63f3ba thru 0xc63fb2e data_size 0x1(0x775 bytes) 2024-09-16T13:29:36.872 INFO:teuthology.orchestra.run.smithi076.stdout:4415 punch from 0x84bfe03 to 0x84ca7ec, (0xa9e9 bytes) 2024-09-16T13:29:36.873 INFO:teuthology.orchestra.run.smithi076.stdout:4417 read 0x66caf76 thru 0x66da647 (0xf6d2 bytes) 2024-09-16T13:29:36.873 INFO:teuthology.orchestra.run.smithi076.stdout:4420 write 0x2e4b911 thru 0x2e59e93 (0xe583 bytes) 2024-09-16T13:29:36.875 INFO:teuthology.orchestra.run.smithi076.stdout:4422 write 0x54e3859 thru 0x54edfd0 (0xa778 bytes) 2024-09-16T13:29:36.888 INFO:teuthology.orchestra.run.smithi076.stdout:4423 write 0xedc2879 thru 0xedcb203 (0x898b bytes) 2024-09-16T13:29:36.892 INFO:teuthology.orchestra.run.smithi076.stdout:4424 write 0x73ba220 thru 0x73c4107 (0x9ee8 bytes) 2024-09-16T13:29:36.898 INFO:teuthology.orchestra.run.smithi076.stdout:4425 trunc from 0xedcb204 to 0xd3783ad 2024-09-16T13:29:36.908 INFO:teuthology.orchestra.run.smithi076.stdout:4426 write 0x88068ea thru 0x8814e11 (0xe528 bytes) 2024-09-16T13:29:36.914 INFO:teuthology.orchestra.run.smithi076.stdout:4427 read 0xbb7ef47 thru 0xbb8d8ca (0xe984 bytes) 2024-09-16T13:29:36.914 INFO:teuthology.orchestra.run.smithi076.stdout:4429 write 0x45cfd79 thru 0x45d30d6 (0x335e bytes) 2024-09-16T13:29:36.919 INFO:teuthology.orchestra.run.smithi076.stdout:4431 write 0x62f8237 thru 0x62f9768 (0x1532 bytes) 2024-09-16T13:29:36.920 INFO:teuthology.orchestra.run.smithi076.stdout:4432 read 0xe072b3 thru 0xe0fe9f (0x8bed bytes) 2024-09-16T13:29:36.920 INFO:teuthology.orchestra.run.smithi076.stdout:4433 write 0x5bedd32 thru 0x5bf78d4 (0x9ba3 bytes) 2024-09-16T13:29:36.925 INFO:teuthology.orchestra.run.smithi076.stdout:4434 writesame 0xa9752a7 thru 0xa982015 data_size 0xb(0xcd6f bytes) 2024-09-16T13:29:36.934 INFO:teuthology.orchestra.run.smithi076.stdout:4436 read 0x206705c thru 0x206c227 (0x51cc bytes) 2024-09-16T13:29:36.934 INFO:teuthology.orchestra.run.smithi076.stdout:4437 write 0xa3461b8 thru 0xa35398b (0xd7d4 bytes) 2024-09-16T13:29:36.942 INFO:teuthology.orchestra.run.smithi076.stdout:4439 read 0x120dc52 thru 0x121bbe4 (0xdf93 bytes) 2024-09-16T13:29:36.943 INFO:teuthology.orchestra.run.smithi076.stdout:4440 write 0x1b1128c thru 0x1b122c6 (0x103b bytes) 2024-09-16T13:29:36.947 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:36.948 INFO:teuthology.orchestra.run.smithi076.stdout:4445 write 0x87881f5 thru 0x8791fb6 (0x9dc2 bytes) 2024-09-16T13:29:36.952 INFO:teuthology.orchestra.run.smithi076.stdout:4446 punch from 0xa1a9b96 to 0xa1b6324, (0xc78e bytes) 2024-09-16T13:29:36.959 INFO:teuthology.orchestra.run.smithi076.stdout:4447 write 0xeb71a6c thru 0xeb72f1c (0x14b1 bytes) 2024-09-16T13:29:36.962 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:36.962 INFO:teuthology.orchestra.run.smithi076.stdout:4450 writesame 0x2c1d0a0 thru 0x2c203b1 data_size 0x3(0x3312 bytes) 2024-09-16T13:29:36.967 INFO:teuthology.orchestra.run.smithi076.stdout:4451 punch from 0xc4d3a3f to 0xc4e20c6, (0xe687 bytes) 2024-09-16T13:29:36.969 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:36.968+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:36.971 INFO:teuthology.orchestra.run.smithi076.stdout:4453 trunc from 0xeb72f1d to 0xb98d937 2024-09-16T13:29:36.991 INFO:teuthology.orchestra.run.smithi076.stdout:4454 trunc from 0xb98d937 to 0x548178e 2024-09-16T13:29:37.017 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:37.017 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:37.018 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:37.018 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:37.022 INFO:teuthology.orchestra.run.smithi076.stdout:4457 trunc from 0x548178e to 0x97f5221 2024-09-16T13:29:37.026 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:37.028 INFO:teuthology.orchestra.run.smithi076.stdout:4458 write 0x1793e39 thru 0x179759e (0x3766 bytes) 2024-09-16T13:29:37.036 INFO:teuthology.orchestra.run.smithi076.stdout:4459 writesame 0x9063591 thru 0x9066d4e data_size 0x5(0x37be bytes) 2024-09-16T13:29:37.041 INFO:teuthology.orchestra.run.smithi076.stdout:4461 write 0x488a263 thru 0x489435a (0xa0f8 bytes) 2024-09-16T13:29:37.046 INFO:teuthology.orchestra.run.smithi076.stdout:4462 writesame 0x9402618 thru 0x9403d05 data_size 0x5(0x16ee bytes) 2024-09-16T13:29:37.052 INFO:teuthology.orchestra.run.smithi076.stdout:4464 trunc from 0x97f5221 to 0xbf0bde3 2024-09-16T13:29:37.069 INFO:teuthology.orchestra.run.smithi076.stdout:4465 trunc from 0xbf0bde3 to 0x1985f13 2024-09-16T13:29:37.106 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.107 INFO:teuthology.orchestra.run.smithi076.stdout:4467 read 0xaacd35 thru 0xaaf2c6 (0x2592 bytes) 2024-09-16T13:29:37.111 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:37.110+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:37.129 INFO:teuthology.orchestra.run.smithi076.stdout:4468 write 0xec36fc1 thru 0xec3bf38 (0x4f78 bytes) 2024-09-16T13:29:37.135 INFO:teuthology.orchestra.run.smithi076.stdout:4471 writesame 0xc8c29af thru 0xc8d22e0 data_size 0x2(0xf932 bytes) 2024-09-16T13:29:37.149 INFO:teuthology.orchestra.run.smithi076.stdout:4472 read 0x35d57be thru 0x35e4749 (0xef8c bytes) 2024-09-16T13:29:37.149 INFO:teuthology.orchestra.run.smithi076.stdout:4476 writesame 0x58c5b5 thru 0x593bb5 data_size 0x1(0x7601 bytes) 2024-09-16T13:29:37.171 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:37.170+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:37.172 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:37.172 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:37.172 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:37.172 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:37.172 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:37.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:37.188 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:37.189 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:29:37.190 INFO:teuthology.orchestra.run.smithi076.stdout:4477 read 0x3d88b12 thru 0x3d89f55 (0x1444 bytes) 2024-09-16T13:29:37.190 INFO:teuthology.orchestra.run.smithi076.stdout:4479 read 0x79be33 thru 0x7a2e57 (0x7025 bytes) 2024-09-16T13:29:37.191 INFO:teuthology.orchestra.run.smithi076.stdout:4481 read 0x7d4576b thru 0x7d4e442 (0x8cd8 bytes) 2024-09-16T13:29:37.191 INFO:teuthology.orchestra.run.smithi076.stdout:4482 punch from 0x10a1200 to 0x10a4c84, (0x3a84 bytes) 2024-09-16T13:29:37.191 INFO:teuthology.orchestra.run.smithi076.stdout:4483 write 0x87c01e8 thru 0x87c3170 (0x2f89 bytes) 2024-09-16T13:29:37.194 INFO:teuthology.orchestra.run.smithi076.stdout:4484 write 0xa5a9d5f thru 0xa5ac92b (0x2bcd bytes) 2024-09-16T13:29:37.198 INFO:teuthology.orchestra.run.smithi076.stdout:4486 read 0x5ecfb77 thru 0x5ed6bb9 (0x7043 bytes) 2024-09-16T13:29:37.198 INFO:teuthology.orchestra.run.smithi076.stdout:4487 write 0x7a1433d thru 0x7a1ed39 (0xa9fd bytes) 2024-09-16T13:29:37.201 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-16T13:29:36.353640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1431181108,"num_objects":1861,"num_object_clones":488,"num_object_copies":3722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":688,"num_read":128019,"num_read_kb":624519,"num_write":15141,"num_write_kb":386768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":55679808,"num_keys_recovered":84,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11490,"ondisk_log_size":11490,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2517248,"kb_used_data":67636,"kb_used_omap":13,"kb_used_meta":2449458,"kb_avail":752457472,"statfs":{"total":773094113280,"available":770516451328,"internally_reserved":0,"allocated":69259264,"data_stored":581949164,"data_compressed":14762694,"data_compressed_allocated":48500736,"data_compressed_original":569835697,"omap_allocated":13632,"internal_metadata":2508245696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":230835529,"num_objects":177,"num_object_clones":61,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":68,"num_read":15300,"num_read_kb":67228,"num_write":1102,"num_write_kb":37942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.021450"},"pg_stats":[{"pgid":"4.1f","version":"86'238","reported_seq":1532,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.584234+0000","last_change":"2024-09-16T13:28:28.989963+0000","last_active":"2024-09-16T13:29:32.584234+0000","last_peered":"2024-09-16T13:29:32.584234+0000","last_clean":"2024-09-16T13:29:32.584234+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:29:32.584234+0000","last_undegraded":"2024-09-16T13:29:32.584234+0000","last_fullsized":"2024-09-16T13:29:32.584234+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089281099999999997,"stat_sum":{"num_bytes":51906101,"num_objects":46,"num_object_clones":11,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":20,"num_read":2280,"num_read_kb":10770,"num_write":238,"num_write_kb":2792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"86'907","reported_seq":10919,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.912863+0000","last_change":"2024-09-16T13:29:09.921994+0000","last_active":"2024-09-16T13:29:31.912863+0000","last_peered":"2024-09-16T13:29:31.912863+0000","last_clean":"2024-09-16T13:29:31.912863+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:29:31.912863+0000","last_undegraded":"2024-09-16T13:29:31.912863+0000","last_fullsized":"2024-09-16T13:29:31.912863+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041021900000000002,"stat_sum":{"num_bytes":58800777,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":20,"num_read":14242,"num_read_kb":23620,"num_write":1503,"num_write_kb":7003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"88'291","reported_seq":974,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.799367+0000","last_change":"2024-09-16T13:28:28.984841+0000","last_active":"2024-09-16T13:29:33.799367+0000","last_peered":"2024-09-16T13:29:33.799367+0000","last_clean":"2024-09-16T13:29:33.799367+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:29:33.799367+0000","last_undegraded":"2024-09-16T13:29:33.799367+0000","last_fullsized":"2024-09-16T13:29:33.799367+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071631099999999997,"stat_sum":{"num_bytes":35952187,"num_objects":47,"num_object_clones":13,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":22,"num_read":1424,"num_read_kb":5686,"num_write":330,"num_write_kb":4479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"87'279","reported_seq":3231,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.584665+0000","last_change":"2024-09-16T13:28:28.981596+0000","last_active":"2024-09-16T13:29:32.584665+0000","last_peered":"2024-09-16T13:29:32.584665+0000","last_clean":"2024-09-16T13:29:32.584665+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:29:32.584665+0000","last_undegraded":"2024-09-16T13:29:32.584665+0000","last_fullsized":"2024-09-16T13:29:32.584665+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030921799999999999,"stat_sum":{"num_bytes":29769192,"num_objects":47,"num_object_clones":12,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":14,"num_read":5803,"num_read_kb":23628,"num_write":285,"num_write_kb":9252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"87'298","reported_seq":1389,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.584036+0000","last_change":"2024-09-16T13:28:28.979438+0000","last_active":"2024-09-16T13:29:32.584036+0000","last_peered":"2024-09-16T13:29:32.584036+0000","last_clean":"2024-09-16T13:29:32.584036+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:29:32.584036+0000","last_undegraded":"2024-09-16T13:29:32.584036+0000","last_fullsized":"2024-09-16T13:29:32.584036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056234500000000001,"stat_sum":{"num_bytes":32346319,"num_objects":52,"num_object_clones":15,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":19,"num_read":1666,"num_read_kb":12900,"num_write":401,"num_write_kb":8106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"88'283","reported_seq":2870,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.156811+0000","last_change":"2024-09-16T13:28:28.979457+0000","last_active":"2024-09-16T13:29:34.156811+0000","last_peered":"2024-09-16T13:29:34.156811+0000","last_clean":"2024-09-16T13:29:34.156811+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:29:34.156811+0000","last_undegraded":"2024-09-16T13:29:34.156811+0000","last_fullsized":"2024-09-16T13:29:34.156811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058420599999999998,"stat_sum":{"num_bytes":40114034,"num_objects":51,"num_object_clones":10,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":16,"num_read":2972,"num_read_kb":10431,"num_write":333,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"88'234","reported_seq":2871,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.530446+0000","last_change":"2024-09-16T13:28:28.983849+0000","last_active":"2024-09-16T13:29:33.530446+0000","last_peered":"2024-09-16T13:29:33.530446+0000","last_clean":"2024-09-16T13:29:33.530446+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:29:33.530446+0000","last_undegraded":"2024-09-16T13:29:33.530446+0000","last_fullsized":"2024-09-16T13:29:33.530446+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041077799999999999,"stat_sum":{"num_bytes":54990339,"num_objects":78,"num_object_clones":22,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":29,"num_read":2979,"num_read_kb":15135,"num_write":270,"num_write_kb":20652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"88'259","reported_seq":2707,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.145903+0000","last_change":"2024-09-16T13:28:28.979247+0000","last_active":"2024-09-16T13:29:34.145903+0000","last_peered":"2024-09-16T13:29:34.145903+0000","last_clean":"2024-09-16T13:29:34.145903+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:29:34.145903+0000","last_undegraded":"2024-09-16T13:29:34.145903+0000","last_fullsized":"2024-09-16T13:29:34.145903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039287600000000001,"stat_sum":{"num_bytes":70034281,"num_objects":63,"num_object_clones":18,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":17,"num_read":3058,"num_read_kb":24815,"num_write":303,"num_write_kb":29443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"88'255","reported_seq":2370,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.017100+0000","last_change":"2024-09-16T13:29:07.285303+0000","last_active":"2024-09-16T13:29:34.017100+0000","last_peered":"2024-09-16T13:29:34.017100+0000","last_clean":"2024-09-16T13:29:34.017100+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:29:34.017100+0000","last_undegraded":"2024-09-16T13:29:34.017100+0000","last_fullsized":"2024-09-16T13:29:34.017100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080754000000000004,"stat_sum":{"num_bytes":38853871,"num_objects":64,"num_object_clones":18,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":30,"num_read":3515,"num_read_kb":24145,"num_write":248,"num_write_kb":14562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"87'386","reported_seq":1453,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.582295+0000","last_change":"2024-09-16T13:28:30.062793+0000","last_active":"2024-09-16T13:29:32.582295+0000","last_peered":"2024-09-16T13:29:32.582295+0000","last_clean":"2024-09-16T13:29:32.582295+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:29:32.582295+0000","last_undegraded":"2024-09-16T13:29:32.582295+0000","last_fullsized":"2024-09-16T13:29:32.582295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039817100000000001,"stat_sum":{"num_bytes":16848701,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":13,"num_read":2924,"num_read_kb":7037,"num_write":480,"num_write_kb":5639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'402","reported_seq":3181,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.048760+0000","last_change":"2024-09-16T13:29:10.916972+0000","last_active":"2024-09-16T13:29:34.048760+0000","last_peered":"2024-09-16T13:29:34.048760+0000","last_clean":"2024-09-16T13:29:34.048760+0000","last_became_active":"2024-09-16T13:29:10.916477+0000","last_became_peered":"2024-09-16T13:29:10.916477+0000","last_unstale":"2024-09-16T13:29:34.048760+0000","last_undegraded":"2024-09-16T13:29:34.048760+0000","last_fullsized":"2024-09-16T13:29:34.048760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:28.629750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025054,"stat_sum":{"num_bytes":77609079,"num_objects":62,"num_object_clones":16,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":21,"num_read":3945,"num_read_kb":26765,"num_write":624,"num_write_kb":13818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"88'343","reported_seq":2524,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.047068+0000","last_change":"2024-09-16T13:28:28.984368+0000","last_active":"2024-09-16T13:29:34.047068+0000","last_peered":"2024-09-16T13:29:34.047068+0000","last_clean":"2024-09-16T13:29:34.047068+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:29:34.047068+0000","last_undegraded":"2024-09-16T13:29:34.047068+0000","last_fullsized":"2024-09-16T13:29:34.047068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000271229,"stat_sum":{"num_bytes":18942505,"num_objects":56,"num_object_clones":14,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":23,"num_read":4092,"num_read_kb":9344,"num_write":399,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"86'221","reported_seq":1386,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.332362+0000","last_change":"2024-09-16T13:28:28.979874+0000","last_active":"2024-09-16T13:29:32.332362+0000","last_peered":"2024-09-16T13:29:32.332362+0000","last_clean":"2024-09-16T13:29:32.332362+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:29:32.332362+0000","last_undegraded":"2024-09-16T13:29:32.332362+0000","last_fullsized":"2024-09-16T13:29:32.332362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000816496,"stat_sum":{"num_bytes":45757587,"num_objects":65,"num_object_clones":19,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":28,"num_read":1255,"num_read_kb":17865,"num_write":237,"num_write_kb":7301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"87'601","reported_seq":3087,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.611697+0000","last_change":"2024-09-16T13:28:28.979687+0000","last_active":"2024-09-16T13:29:32.611697+0000","last_peered":"2024-09-16T13:29:32.611697+0000","last_clean":"2024-09-16T13:29:32.611697+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:29:32.611697+0000","last_undegraded":"2024-09-16T13:29:32.611697+0000","last_fullsized":"2024-09-16T13:29:32.611697+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035537999999999998,"stat_sum":{"num_bytes":14427395,"num_objects":47,"num_object_clones":10,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":19,"num_read":7100,"num_read_kb":12632,"num_write":814,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"88'349","reported_seq":3577,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.992675+0000","last_change":"2024-09-16T13:29:09.920433+0000","last_active":"2024-09-16T13:29:33.992675+0000","last_peered":"2024-09-16T13:29:33.992675+0000","last_clean":"2024-09-16T13:29:33.992675+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:29:33.992675+0000","last_undegraded":"2024-09-16T13:29:33.992675+0000","last_fullsized":"2024-09-16T13:29:33.992675+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035811000000000003,"stat_sum":{"num_bytes":47185222,"num_objects":71,"num_object_clones":24,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":29,"num_read":3420,"num_read_kb":8589,"num_write":422,"num_write_kb":20967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"88'561","reported_seq":2818,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.901832+0000","last_change":"2024-09-16T13:28:28.983418+0000","last_active":"2024-09-16T13:29:33.901832+0000","last_peered":"2024-09-16T13:29:33.901832+0000","last_clean":"2024-09-16T13:29:33.901832+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:29:33.901832+0000","last_undegraded":"2024-09-16T13:29:33.901832+0000","last_fullsized":"2024-09-16T13:29:33.901832+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044577700000000001,"stat_sum":{"num_bytes":28268662,"num_objects":44,"num_object_clones":9,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":18,"num_read":6892,"num_read_kb":16598,"num_write":843,"num_write_kb":8330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.325471+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:29:31.325471+0000","last_peered":"2024-09-16T13:29:31.325471+0000","last_clean":"2024-09-16T13:29:31.325471+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:29:31.325471+0000","last_undegraded":"2024-09-16T13:29:31.325471+0000","last_fullsized":"2024-09-16T13:29:31.325471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"87'354","reported_seq":3087,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.095769+0000","last_change":"2024-09-16T13:28:28.979658+0000","last_active":"2024-09-16T13:29:34.095769+0000","last_peered":"2024-09-16T13:29:34.095769+0000","last_clean":"2024-09-16T13:29:34.095769+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:29:34.095769+0000","last_undegraded":"2024-09-16T13:29:34.095769+0000","last_fullsized":"2024-09-16T13:29:34.095769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066082900000000004,"stat_sum":{"num_bytes":41949180,"num_objects":72,"num_object_clones":25,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":30,"num_read":3313,"num_read_kb":34038,"num_write":499,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":79,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.318830+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:31.318830+0000","last_peered":"2024-09-16T13:29:31.318830+0000","last_clean":"2024-09-16T13:29:31.318830+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:29:31.318830+0000","last_undegraded":"2024-09-16T13:29:31.318830+0000","last_fullsized":"2024-09-16T13:29:31.318830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"86'240","reported_seq":2235,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.333625+0000","last_change":"2024-09-16T13:29:10.918359+0000","last_active":"2024-09-16T13:29:33.333625+0000","last_peered":"2024-09-16T13:29:33.333625+0000","last_clean":"2024-09-16T13:29:33.333625+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:29:33.333625+0000","last_undegraded":"2024-09-16T13:29:33.333625+0000","last_fullsized":"2024-09-16T13:29:33.333625+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043631399999999998,"stat_sum":{"num_bytes":32268823,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":23,"num_read":2235,"num_read_kb":14333,"num_write":343,"num_write_kb":6161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.325459+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:29:31.325459+0000","last_peered":"2024-09-16T13:29:31.325459+0000","last_clean":"2024-09-16T13:29:31.325459+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:29:31.325459+0000","last_undegraded":"2024-09-16T13:29:31.325459+0000","last_fullsized":"2024-09-16T13:29:31.325459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"85'407","reported_seq":2882,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.750882+0000","last_change":"2024-09-16T13:28:30.062555+0000","last_active":"2024-09-16T13:29:31.750882+0000","last_peered":"2024-09-16T13:29:31.750882+0000","last_clean":"2024-09-16T13:29:31.750882+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:29:31.750882+0000","last_undegraded":"2024-09-16T13:29:31.750882+0000","last_fullsized":"2024-09-16T13:29:31.750882+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000230199,"stat_sum":{"num_bytes":58398808,"num_objects":70,"num_object_clones":21,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":24,"num_read":3438,"num_read_kb":44855,"num_write":538,"num_write_kb":27737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.318695+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:31.318695+0000","last_peered":"2024-09-16T13:29:31.318695+0000","last_clean":"2024-09-16T13:29:31.318695+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:29:31.318695+0000","last_undegraded":"2024-09-16T13:29:31.318695+0000","last_fullsized":"2024-09-16T13:29:31.318695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"86'171","reported_seq":1671,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.959837+0000","last_change":"2024-09-16T13:28:28.979887+0000","last_active":"2024-09-16T13:29:31.959837+0000","last_peered":"2024-09-16T13:29:31.959837+0000","last_clean":"2024-09-16T13:29:31.959837+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:29:31.959837+0000","last_undegraded":"2024-09-16T13:29:31.959837+0000","last_fullsized":"2024-09-16T13:29:31.959837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053138000000000003,"stat_sum":{"num_bytes":17592455,"num_objects":45,"num_object_clones":10,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":11,"num_read":2645,"num_read_kb":11753,"num_write":187,"num_write_kb":22497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.332382+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:29:32.332382+0000","last_peered":"2024-09-16T13:29:32.332382+0000","last_clean":"2024-09-16T13:29:32.332382+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:29:32.332382+0000","last_undegraded":"2024-09-16T13:29:32.332382+0000","last_fullsized":"2024-09-16T13:29:32.332382+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.554224+0000","last_change":"2024-09-16T13:28:12.317911+0000","last_active":"2024-09-16T13:29:32.554224+0000","last_peered":"2024-09-16T13:29:32.554224+0000","last_clean":"2024-09-16T13:29:32.554224+0000","last_became_active":"2024-09-16T13:28:10.309136+0000","last_became_peered":"2024-09-16T13:28:10.309136+0000","last_unstale":"2024-09-16T13:29:32.554224+0000","last_undegraded":"2024-09-16T13:29:32.554224+0000","last_fullsized":"2024-09-16T13:29:32.554224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:50.769889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"87'213","reported_seq":3303,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.454853+0000","last_change":"2024-09-16T13:28:28.979855+0000","last_active":"2024-09-16T13:29:32.454853+0000","last_peered":"2024-09-16T13:29:32.454853+0000","last_clean":"2024-09-16T13:29:32.454853+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:29:32.454853+0000","last_undegraded":"2024-09-16T13:29:32.454853+0000","last_fullsized":"2024-09-16T13:29:32.454853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039579799999999999,"stat_sum":{"num_bytes":34126827,"num_objects":44,"num_object_clones":13,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":15,"num_read":3333,"num_read_kb":23121,"num_write":256,"num_write_kb":4925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.318909+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:29:31.318909+0000","last_peered":"2024-09-16T13:29:31.318909+0000","last_clean":"2024-09-16T13:29:31.318909+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:29:31.318909+0000","last_undegraded":"2024-09-16T13:29:31.318909+0000","last_fullsized":"2024-09-16T13:29:31.318909+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":79,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.318350+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:29:31.318350+0000","last_peered":"2024-09-16T13:29:31.318350+0000","last_clean":"2024-09-16T13:29:31.318350+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:29:31.318350+0000","last_undegraded":"2024-09-16T13:29:31.318350+0000","last_fullsized":"2024-09-16T13:29:31.318350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"87'302","reported_seq":2162,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.566956+0000","last_change":"2024-09-16T13:28:28.980209+0000","last_active":"2024-09-16T13:29:32.566956+0000","last_peered":"2024-09-16T13:29:32.566956+0000","last_clean":"2024-09-16T13:29:32.566956+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:29:32.566956+0000","last_undegraded":"2024-09-16T13:29:32.566956+0000","last_fullsized":"2024-09-16T13:29:32.566956+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069467499999999996,"stat_sum":{"num_bytes":59265421,"num_objects":69,"num_object_clones":20,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":24,"num_read":3259,"num_read_kb":11250,"num_write":351,"num_write_kb":5273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"53'1","reported_seq":71,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.354882+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:29:31.354882+0000","last_peered":"2024-09-16T13:29:31.354882+0000","last_clean":"2024-09-16T13:29:31.354882+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:29:31.354882+0000","last_undegraded":"2024-09-16T13:29:31.354882+0000","last_fullsized":"2024-09-16T13:29:31.354882+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.563121+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:29:32.563121+0000","last_peered":"2024-09-16T13:29:32.563121+0000","last_clean":"2024-09-16T13:29:32.563121+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:29:32.563121+0000","last_undegraded":"2024-09-16T13:29:32.563121+0000","last_fullsized":"2024-09-16T13:29:32.563121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"86'287","reported_seq":2598,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.912098+0000","last_change":"2024-09-16T13:28:30.062973+0000","last_active":"2024-09-16T13:29:31.912098+0000","last_peered":"2024-09-16T13:29:31.912098+0000","last_clean":"2024-09-16T13:29:31.912098+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:29:31.912098+0000","last_undegraded":"2024-09-16T13:29:31.912098+0000","last_fullsized":"2024-09-16T13:29:31.912098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049121500000000003,"stat_sum":{"num_bytes":94433618,"num_objects":66,"num_object_clones":17,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":27,"num_read":3373,"num_read_kb":36546,"num_write":339,"num_write_kb":28954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"53'1","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.338752+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:29:32.338752+0000","last_peered":"2024-09-16T13:29:32.338752+0000","last_clean":"2024-09-16T13:29:32.338752+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:29:32.338752+0000","last_undegraded":"2024-09-16T13:29:32.338752+0000","last_fullsized":"2024-09-16T13:29:32.338752+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:20:30.803710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.554279+0000","last_change":"2024-09-16T13:28:12.317404+0000","last_active":"2024-09-16T13:29:32.554279+0000","last_peered":"2024-09-16T13:29:32.554279+0000","last_clean":"2024-09-16T13:29:32.554279+0000","last_became_active":"2024-09-16T13:28:10.311971+0000","last_became_peered":"2024-09-16T13:28:10.311971+0000","last_unstale":"2024-09-16T13:29:32.554279+0000","last_undegraded":"2024-09-16T13:29:32.554279+0000","last_fullsized":"2024-09-16T13:29:32.554279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:58.394506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.563134+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:29:32.563134+0000","last_peered":"2024-09-16T13:29:32.563134+0000","last_clean":"2024-09-16T13:29:32.563134+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:29:32.563134+0000","last_undegraded":"2024-09-16T13:29:32.563134+0000","last_fullsized":"2024-09-16T13:29:32.563134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"85'445","reported_seq":1814,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.744738+0000","last_change":"2024-09-16T13:28:28.989279+0000","last_active":"2024-09-16T13:29:31.744738+0000","last_peered":"2024-09-16T13:29:31.744738+0000","last_clean":"2024-09-16T13:29:31.744738+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:29:31.744738+0000","last_undegraded":"2024-09-16T13:29:31.744738+0000","last_fullsized":"2024-09-16T13:29:31.744738+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024121900000000001,"stat_sum":{"num_bytes":30560340,"num_objects":57,"num_object_clones":13,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":20,"num_read":3017,"num_read_kb":14691,"num_write":616,"num_write_kb":6770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"88'716","reported_seq":4758,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.150611+0000","last_change":"2024-09-16T13:29:10.918865+0000","last_active":"2024-09-16T13:29:34.150611+0000","last_peered":"2024-09-16T13:29:34.150611+0000","last_clean":"2024-09-16T13:29:34.150611+0000","last_became_active":"2024-09-16T13:29:10.916898+0000","last_became_peered":"2024-09-16T13:29:10.916898+0000","last_unstale":"2024-09-16T13:29:34.150611+0000","last_undegraded":"2024-09-16T13:29:34.150611+0000","last_fullsized":"2024-09-16T13:29:34.150611+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:07.302247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024448900000000002,"stat_sum":{"num_bytes":39813005,"num_objects":57,"num_object_clones":13,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":15,"num_read":7210,"num_read_kb":23229,"num_write":1058,"num_write_kb":10454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"88'258","reported_seq":4171,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.099265+0000","last_change":"2024-09-16T13:28:28.979223+0000","last_active":"2024-09-16T13:29:34.099265+0000","last_peered":"2024-09-16T13:29:34.099265+0000","last_clean":"2024-09-16T13:29:34.099265+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:29:34.099265+0000","last_undegraded":"2024-09-16T13:29:34.099265+0000","last_fullsized":"2024-09-16T13:29:34.099265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039311499999999998,"stat_sum":{"num_bytes":76610389,"num_objects":62,"num_object_clones":20,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":18,"num_read":5264,"num_read_kb":28181,"num_write":348,"num_write_kb":8942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"87'311","reported_seq":3568,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-16T13:29:32.367635+0000","last_change":"2024-09-16T13:28:28.990059+0000","last_active":"2024-09-16T13:29:32.367635+0000","last_peered":"2024-09-16T13:29:32.367635+0000","last_clean":"2024-09-16T13:29:32.367635+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:29:32.367635+0000","last_undegraded":"2024-09-16T13:29:32.367635+0000","last_fullsized":"2024-09-16T13:29:32.367635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086978299999999995,"stat_sum":{"num_bytes":47448741,"num_objects":55,"num_object_clones":14,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":20,"num_read":4584,"num_read_kb":42098,"num_write":383,"num_write_kb":24712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"86'219","reported_seq":4561,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.866785+0000","last_change":"2024-09-16T13:28:30.062577+0000","last_active":"2024-09-16T13:29:31.866785+0000","last_peered":"2024-09-16T13:29:31.866785+0000","last_clean":"2024-09-16T13:29:31.866785+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:29:31.866785+0000","last_undegraded":"2024-09-16T13:29:31.866785+0000","last_fullsized":"2024-09-16T13:29:31.866785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029103499999999998,"stat_sum":{"num_bytes":87349127,"num_objects":53,"num_object_clones":14,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":17,"num_read":4885,"num_read_kb":31255,"num_write":286,"num_write_kb":22312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"88'345","reported_seq":4937,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.752299+0000","last_change":"2024-09-16T13:28:28.979711+0000","last_active":"2024-09-16T13:29:33.752299+0000","last_peered":"2024-09-16T13:29:33.752299+0000","last_clean":"2024-09-16T13:29:33.752299+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:29:33.752299+0000","last_undegraded":"2024-09-16T13:29:33.752299+0000","last_fullsized":"2024-09-16T13:29:33.752299+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071623100000000001,"stat_sum":{"num_bytes":51073083,"num_objects":62,"num_object_clones":13,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":23,"num_read":6168,"num_read_kb":24077,"num_write":448,"num_write_kb":7795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"88'334","reported_seq":2308,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:34.157185+0000","last_change":"2024-09-16T13:28:28.979910+0000","last_active":"2024-09-16T13:29:34.157185+0000","last_peered":"2024-09-16T13:29:34.157185+0000","last_clean":"2024-09-16T13:29:34.157185+0000","last_became_active":"2024-09-16T13:28:26.987044+0000","last_became_peered":"2024-09-16T13:28:26.987044+0000","last_unstale":"2024-09-16T13:29:34.157185+0000","last_undegraded":"2024-09-16T13:29:34.157185+0000","last_fullsized":"2024-09-16T13:29:34.157185+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:11.809480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084175000000000005,"stat_sum":{"num_bytes":44676439,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":28,"num_read":2397,"num_read_kb":10955,"num_write":375,"num_write_kb":8732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"88'273","reported_seq":1309,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-16T13:29:33.992888+0000","last_change":"2024-09-16T13:28:28.980116+0000","last_active":"2024-09-16T13:29:33.992888+0000","last_peered":"2024-09-16T13:29:33.992888+0000","last_clean":"2024-09-16T13:29:33.992888+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:29:33.992888+0000","last_undegraded":"2024-09-16T13:29:33.992888+0000","last_fullsized":"2024-09-16T13:29:33.992888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010142790000000001,"stat_sum":{"num_bytes":24917478,"num_objects":58,"num_object_clones":12,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":27,"num_read":1531,"num_read_kb":15207,"num_write":346,"num_write_kb":9630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"86'631","reported_seq":2249,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-16T13:29:31.531363+0000","last_change":"2024-09-16T13:28:58.119992+0000","last_active":"2024-09-16T13:29:31.531363+0000","last_peered":"2024-09-16T13:29:31.531363+0000","last_clean":"2024-09-16T13:29:31.531363+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:29:31.531363+0000","last_undegraded":"2024-09-16T13:29:31.531363+0000","last_fullsized":"2024-09-16T13:29:31.531363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074187000000000005,"stat_sum":{"num_bytes":28300735,"num_objects":59,"num_object_clones":17,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":26,"num_read":3734,"num_read_kb":8915,"num_write":935,"num_write_kb":5850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1430590721,"num_objects":1855,"num_object_clones":488,"num_object_copies":3710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":686,"num_read":127953,"num_read_kb":624464,"num_write":15038,"num_write_kb":385138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":54499072,"num_keys_recovered":84,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61845504,"data_stored":587854931,"data_compressed":14909404,"data_compressed_allocated":49238016,"data_compressed_original":579698865,"omap_allocated":0,"internal_metadata":0},"log_size":11413,"ondisk_log_size":11413,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316484,"kb_used_data":6648,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94055356,"statfs":{"total":96636764160,"available":96312684544,"internally_reserved":0,"allocated":6807552,"data_stored":49591241,"data_compressed":1268590,"data_compressed_allocated":4587520,"data_compressed_original":48344566,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.248}]},{"osd":2,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.502},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.434}]},{"osd":3,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.869,"5min":1.869,"15min":1.869},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":1.2070000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.35999999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.374}]}]},{"osd":5,"up_from":15,"seq":64424509464,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318728,"kb_used_data":8888,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94053112,"statfs":{"total":96636764160,"available":96310386688,"internally_reserved":0,"allocated":9101312,"data_stored":82036310,"data_compressed":1843233,"data_compressed_allocated":6488064,"data_compressed_original":80462472,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.312},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.169,"5min":2.169,"15min":2.169},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.498},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.43099999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.54800000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320004,"kb_used_data":10168,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94051836,"statfs":{"total":96636764160,"available":96309080064,"internally_reserved":0,"allocated":10412032,"data_stored":77277730,"data_compressed":2688187,"data_compressed_allocated":7348224,"data_compressed_original":75401107,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.252},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.377}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.442},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39700000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509462,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306448,"kb_used_data":9044,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94065392,"statfs":{"total":96636764160,"available":96322961408,"internally_reserved":0,"allocated":9261056,"data_stored":61492483,"data_compressed":1979504,"data_compressed_allocated":6053888,"data_compressed_original":59634873,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.38}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.33600000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.622},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51500000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.55700000000000005}]}]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316736,"kb_used_data":6904,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94055104,"statfs":{"total":96636764160,"available":96312426496,"internally_reserved":0,"allocated":7069696,"data_stored":67106327,"data_compressed":1467949,"data_compressed_allocated":4976640,"data_compressed_original":65949360,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.371},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.28999999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.34000000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309940,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061900,"statfs":{"total":96636764160,"available":96319385600,"internally_reserved":0,"allocated":4366336,"data_stored":25695157,"data_compressed":913078,"data_compressed_allocated":2396160,"data_compressed_original":24593052,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.224},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.3390000000000004,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.249},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":9.4480000000000004,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.23599999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.371}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.4090000000000007,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.38400000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.432}]},{"osd":6,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.41499999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.5640000000000001,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.443},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.45300000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310564,"kb_used_data":13212,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94061276,"statfs":{"total":96636764160,"available":96318746624,"internally_reserved":0,"allocated":13529088,"data_stored":124667985,"data_compressed":3058034,"data_compressed_allocated":10088448,"data_compressed_original":122618353,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.252}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.312},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.442},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.52700000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.53700000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318344,"kb_used_data":8508,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94053496,"statfs":{"total":96636764160,"available":96310779904,"internally_reserved":0,"allocated":8712192,"data_stored":94081931,"data_compressed":1544119,"data_compressed_allocated":6561792,"data_compressed_original":92831914,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.22},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.249}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.376},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.33400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.36899999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7634944,"data_stored":92996544,"data_compressed":1538520,"data_compressed_allocated":6524928,"data_compressed_original":92258474,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":124172947,"data_compressed":3058034,"data_compressed_allocated":10088448,"data_compressed_original":122618353,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3346432,"data_stored":25200138,"data_compressed":913078,"data_compressed_allocated":2396160,"data_compressed_original":24593052,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8630272,"data_stored":66506984,"data_compressed":2026930,"data_compressed_allocated":6430720,"data_compressed_original":65139897,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9641984,"data_stored":80410573,"data_compressed":2722990,"data_compressed_allocated":7589888,"data_compressed_original":79017875,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8118272,"data_stored":81587504,"data_compressed":1872902,"data_compressed_allocated":6520832,"data_compressed_original":80507528,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":66611308,"data_compressed":1467949,"data_compressed_allocated":4976640,"data_compressed_original":65949360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5918720,"data_stored":50368933,"data_compressed":1309001,"data_compressed_allocated":4710400,"data_compressed_original":49614326,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:29:37.204 INFO:tasks.thrashosds.thrasher:Setting 4.d to [5, 0] 2024-09-16T13:29:37.204 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.d', '5', '0'] 2024-09-16T13:29:37.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.d 5 0 2024-09-16T13:29:37.216 INFO:teuthology.orchestra.run.smithi076.stdout:4488 trunc from 0xec3bf39 to 0x3932e70 2024-09-16T13:29:37.227 INFO:teuthology.orchestra.run.smithi076.stdout:4490 read 0x381bf75 thru 0x3824277 (0x8303 bytes) 2024-09-16T13:29:37.227 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.227 INFO:teuthology.orchestra.run.smithi076.stdout:4493 read 0x2bca27e thru 0x2bd718b (0xcf0e bytes) 2024-09-16T13:29:37.227 INFO:teuthology.orchestra.run.smithi076.stdout:4494 read 0x2e7d24f thru 0x2e81b53 (0x4905 bytes) 2024-09-16T13:29:37.236 INFO:teuthology.orchestra.run.smithi076.stdout:4495 trunc from 0x3932e70 to 0xb8cf50d 2024-09-16T13:29:37.241 INFO:teuthology.orchestra.run.smithi076.stdout:4496 writesame 0x351efef thru 0x35241c7 data_size 0x1(0x51d9 bytes) 2024-09-16T13:29:37.272 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:37.271+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:37.278 INFO:teuthology.orchestra.run.smithi076.stdout:4497 read 0x19fd06f thru 0x1a06138 (0x90ca bytes) 2024-09-16T13:29:37.279 INFO:teuthology.orchestra.run.smithi076.stdout:4498 write 0x6113ca6 thru 0x611e991 (0xacec bytes) 2024-09-16T13:29:37.284 INFO:teuthology.orchestra.run.smithi076.stdout:4499 writesame 0x137a553 thru 0x137a8ef data_size 0x5(0x39d bytes) 2024-09-16T13:29:37.290 INFO:teuthology.orchestra.run.smithi076.stdout:4500 write 0x10c1908 thru 0x10c2756 (0xe4f bytes) 2024-09-16T13:29:37.304 INFO:teuthology.orchestra.run.smithi076.stdout:4501 write 0xdd5d1cc thru 0xdd6ce62 (0xfc97 bytes) 2024-09-16T13:29:37.309 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.309 INFO:teuthology.orchestra.run.smithi076.stdout:4503 writesame 0x2dcd7c6 thru 0x2ddc503 data_size 0x1(0xed3e bytes) 2024-09-16T13:29:37.331 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:37.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.039462241999999, 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:37.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:37.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:37.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:37.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.338 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337814447, 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077160969999999999, 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 44 2024-09-16T13:29:37.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990848+0000", 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990851+0000", 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9189999999999999e-06 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990975+0000", 2024-09-16T13:29:37.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000124012 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990986+0000", 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0275000000000001e-05 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.990990+0000", 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5260000000000004e-06 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993572+0000", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002581664 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993650+0000", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8059000000000006e-05 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.993804+0000", 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015420700000000001 2024-09-16T13:29:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997303+0000", 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034989660000000001 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998488+0000", 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001185221 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998530+0000", 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2245e-05 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998564+0000", 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4003e-05 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337651833999999, 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077934930000000003, 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 46 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.346 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991011+0000", 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1199999999999999e-07 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991016+0000", 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991020+0000", 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.941e-06 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991023+0000", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6840000000000001e-06 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.994975+0000", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039516849999999999 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995028+0000", 2024-09-16T13:29:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3736000000000002e-05 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995111+0000", 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2348000000000005e-05 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997311+0000", 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021999049999999998 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998716+0000", 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014054029999999999 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998758+0000", 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1560000000000002e-05 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998804+0000", 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6774000000000003e-05 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337630786999995, 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078608740000000003, 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.352 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 47 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991032+0000", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7000000000000001e-07 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991104+0000", 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1381e-05 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991112+0000", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1000000000000004e-06 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991115+0000", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2619999999999999e-06 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995574+0000", 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044584719999999998 2024-09-16T13:29:37.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995627+0000", 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.995708+0000", 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1058999999999998e-05 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998247+0000", 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538256 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998845+0000", 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00059833399999999995 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998873+0000", 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8274000000000001e-05 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998893+0000", 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9553000000000001e-05 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337543307000004, 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:37.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:37.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:37.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.367 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:37.368 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337520798, 2024-09-16T13:29:37.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:37.368 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.368 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.368 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:37.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:37.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:37.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:37.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:37.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:37.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.380 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337474470999993, 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:37.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337307714000005, 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:37.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:37.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.337292949000002, 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:37.392 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:37.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:37.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.991970632000005, 2024-09-16T13:29:37.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:37.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.405 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.991932954000006, 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.406 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:37.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:37.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.991910141000005, 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.413 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:37.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:37.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:37.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:37.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.991824928, 2024-09-16T13:29:37.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:37.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:37.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:37.426 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.685246003000003, 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:37.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.659337768999997, 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:37.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:37.435 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 62.590572614000003, 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:37.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 28.089029796999998, 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:37.440 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:37.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:37.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.447 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.894104477999999, 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:29:37.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:29:37.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.452 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.7382274429999995, 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:29:37.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:29:37.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.459275613, 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:37.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:37.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:29:37.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:29:37.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:37.461 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:37.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:29:37.464 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:37.461+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:37.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:37.465 INFO:teuthology.orchestra.run.smithi076.stdout:4506 read 0x6c97a3c thru 0x6ca3cd8 (0xc29d bytes) 2024-09-16T13:29:37.465 INFO:teuthology.orchestra.run.smithi076.stdout:4507 trunc from 0xdd6ce63 to 0x99762ea 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4508 write 0x6e44f6f thru 0x6e50000 (0xb092 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4510 write 0x6c87fd1 thru 0x6c94395 (0xc3c5 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4511 write 0x8317584 thru 0x8320cca (0x9747 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4512 write 0xe7f39ed thru 0xe7fa2f3 (0x6907 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4513 trunc from 0xe7fa2f4 to 0x9c0c548 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4514 write 0xe427 thru 0x1c599 (0xe173 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4515 write 0x48234f7 thru 0x4828ae2 (0x55ec bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4516 write 0x64af312 thru 0x64b8b91 (0x9880 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4517 read 0x71f02af thru 0x71f718c (0x6ede bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4519 read 0x8492b46 thru 0x8498b3e (0x5ff9 bytes) 2024-09-16T13:29:37.466 INFO:teuthology.orchestra.run.smithi076.stdout:4521 writesame 0x80cfc6f thru 0x80d8ab1 data_size 0x1(0x8e43 bytes) 2024-09-16T13:29:37.490 INFO:teuthology.orchestra.run.smithi076.stdout:4522 write 0x7f38022 thru 0x7f3e536 (0x6515 bytes) 2024-09-16T13:29:37.495 INFO:teuthology.orchestra.run.smithi076.stdout:4523 read 0x463ff91 thru 0x464dd51 (0xddc1 bytes) 2024-09-16T13:29:37.496 INFO:teuthology.orchestra.run.smithi076.stdout:4524 read 0x41de88a thru 0x41eb5f1 (0xcd68 bytes) 2024-09-16T13:29:37.496 INFO:teuthology.orchestra.run.smithi076.stdout:4525 read 0x66024c0 thru 0x66037b6 (0x12f7 bytes) 2024-09-16T13:29:37.496 INFO:teuthology.orchestra.run.smithi076.stdout:4526 read 0x42266be thru 0x4231f04 (0xb847 bytes) 2024-09-16T13:29:37.496 INFO:teuthology.orchestra.run.smithi076.stdout:4527 read 0x80c6ebb thru 0x80d515b (0xe2a1 bytes) 2024-09-16T13:29:37.497 INFO:teuthology.orchestra.run.smithi076.stdout:4528 read 0x69258cc thru 0x69331a5 (0xd8da bytes) 2024-09-16T13:29:37.497 INFO:teuthology.orchestra.run.smithi076.stdout:4529 punch from 0x607cade to 0x607d917, (0xe39 bytes) 2024-09-16T13:29:37.497 INFO:teuthology.orchestra.run.smithi076.stdout:4530 write 0x64841b thru 0x650cba (0x88a0 bytes) 2024-09-16T13:29:37.515 INFO:teuthology.orchestra.run.smithi076.stdout:4532 trunc from 0x9c0c548 to 0x4217c9f 2024-09-16T13:29:37.531 INFO:teuthology.orchestra.run.smithi076.stdout:4533 write 0x356aad1 thru 0x3578c60 (0xe190 bytes) 2024-09-16T13:29:37.537 INFO:teuthology.orchestra.run.smithi076.stdout:4534 read 0xeead63 thru 0xef8e66 (0xe104 bytes) 2024-09-16T13:29:37.537 INFO:teuthology.orchestra.run.smithi076.stdout:4537 read 0x2998b1a thru 0x29a1fd7 (0x94be bytes) 2024-09-16T13:29:37.537 INFO:teuthology.orchestra.run.smithi076.stdout:4538 read 0x1704077 thru 0x170cfc1 (0x8f4b bytes) 2024-09-16T13:29:37.546 INFO:teuthology.orchestra.run.smithi076.stdout:4539 write 0x5faae43 thru 0x5face9d (0x205b bytes) 2024-09-16T13:29:37.560 INFO:teuthology.orchestra.run.smithi076.stdout:4540 write 0x9e7134b thru 0x9e72321 (0xfd7 bytes) 2024-09-16T13:29:37.563 INFO:teuthology.orchestra.run.smithi076.stdout:4541 read 0xa0630 thru 0xa680d (0x61de bytes) 2024-09-16T13:29:37.564 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:37.561+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:37.564 INFO:teuthology.orchestra.run.smithi076.stdout:4542 punch from 0x4cfc69 to 0x4d6863, (0x6bfa bytes) 2024-09-16T13:29:37.564 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.573 INFO:teuthology.orchestra.run.smithi076.stdout:4545 trunc from 0x9e72322 to 0x307c306 2024-09-16T13:29:37.585 INFO:teuthology.orchestra.run.smithi076.stdout:4546 trunc from 0x307c306 to 0x3567120 2024-09-16T13:29:37.591 INFO:teuthology.orchestra.run.smithi076.stdout:4548 read 0x1f6f2ea thru 0x1f70289 (0xfa0 bytes) 2024-09-16T13:29:37.591 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:37.591 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:37.591 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:37.592 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:37.602 INFO:teuthology.orchestra.run.smithi076.stdout:4549 trunc from 0x3567120 to 0xcc0657a 2024-09-16T13:29:37.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:37.609 INFO:teuthology.orchestra.run.smithi076.stdout:4550 trunc from 0xcc0657a to 0xbee416d 2024-09-16T13:29:37.614 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.614 INFO:teuthology.orchestra.run.smithi076.stdout:4554 writesame 0x55955c5 thru 0x559cc02 data_size 0x6(0x763e bytes) 2024-09-16T13:29:37.622 INFO:teuthology.orchestra.run.smithi076.stdout:4558 trunc from 0xbee416d to 0x932b631 2024-09-16T13:29:37.628 INFO:teuthology.orchestra.run.smithi076.stdout:4559 read 0x4cdc46e thru 0x4ce3af3 (0x7686 bytes) 2024-09-16T13:29:37.628 INFO:teuthology.orchestra.run.smithi076.stdout:4560 read 0x7c4c710 thru 0x7c597ae (0xd09f bytes) 2024-09-16T13:29:37.628 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.632 INFO:teuthology.orchestra.run.smithi076.stdout:4564 trunc from 0x932b631 to 0x6130684 2024-09-16T13:29:37.637 INFO:teuthology.orchestra.run.smithi076.stdout:4565 read 0x545b15 thru 0x547bd5 (0x20c1 bytes) 2024-09-16T13:29:37.638 INFO:teuthology.orchestra.run.smithi076.stdout:4566 write 0x1b7d10d thru 0x1b815af (0x44a3 bytes) 2024-09-16T13:29:37.649 INFO:teuthology.orchestra.run.smithi076.stdout:4568 write 0x754bb3b thru 0x755053a (0x4a00 bytes) 2024-09-16T13:29:37.653 INFO:teuthology.orchestra.run.smithi076.stdout:4569 punch from 0x41ac507 to 0x41b64ab, (0x9fa4 bytes) 2024-09-16T13:29:37.664 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:37.662+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:37.665 INFO:teuthology.orchestra.run.smithi076.stdout:4570 write 0xdcd95af thru 0xdcdc46f (0x2ec1 bytes) 2024-09-16T13:29:37.669 INFO:teuthology.orchestra.run.smithi076.stdout:4571 read 0x86b627 thru 0x86c542 (0xf1c bytes) 2024-09-16T13:29:37.669 INFO:teuthology.orchestra.run.smithi076.stdout:4572 writesame 0x1648b0f thru 0x1648dac data_size 0x5(0x29e bytes) 2024-09-16T13:29:37.670 INFO:teuthology.orchestra.run.smithi076.stdout:4574 read 0x394c1c9 thru 0x395924c (0xd084 bytes) 2024-09-16T13:29:37.670 INFO:teuthology.orchestra.run.smithi076.stdout:4575 punch from 0x7a1f26 to 0x7ad1d8, (0xb2b2 bytes) 2024-09-16T13:29:37.670 INFO:teuthology.orchestra.run.smithi076.stdout:4576 write 0x4bdcef0 thru 0x4be2220 (0x5331 bytes) 2024-09-16T13:29:37.674 INFO:teuthology.orchestra.run.smithi076.stdout:4579 read 0x588f139 thru 0x5895801 (0x66c9 bytes) 2024-09-16T13:29:37.681 INFO:teuthology.orchestra.run.smithi076.stdout:4581 trunc from 0xdcdc470 to 0x926f089 2024-09-16T13:29:37.691 INFO:teuthology.orchestra.run.smithi076.stdout:4582 read 0x3a94e61 thru 0x3a973ac (0x254c bytes) 2024-09-16T13:29:37.691 INFO:teuthology.orchestra.run.smithi076.stdout:4583 read 0x8717629 thru 0x8721f60 (0xa938 bytes) 2024-09-16T13:29:37.693 INFO:teuthology.orchestra.run.smithi076.stdout:4584 trunc from 0x926f089 to 0xb50048a 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:4586 read 0xb201b7d thru 0xb2048fc (0x2d80 bytes) 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:4587 read 0x60c18db thru 0x60c4191 (0x28b7 bytes) 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:4590 punch from 0x26a3889 to 0x26a8be1, (0x5358 bytes) 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:4592 read 0xac398e5 thru 0xac444eb (0xac07 bytes) 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:4594 read 0x9209372 thru 0x92189c9 (0xf658 bytes) 2024-09-16T13:29:37.699 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.700 INFO:teuthology.orchestra.run.smithi076.stdout:4598 writesame 0x6f2c091 thru 0x6f3b504 data_size 0x95(0xf474 bytes) 2024-09-16T13:29:37.712 INFO:teuthology.orchestra.run.smithi076.stdout:4599 writesame 0xc01f543 thru 0xc02c131 data_size 0x25(0xcbef bytes) 2024-09-16T13:29:37.712 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.712 INFO:teuthology.orchestra.run.smithi076.stdout:4601 write 0x74c765 thru 0x75206c (0x5908 bytes) 2024-09-16T13:29:37.722 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.722 INFO:teuthology.orchestra.run.smithi076.stdout:4603 write 0xbad7957 thru 0xbae3279 (0xb923 bytes) 2024-09-16T13:29:37.727 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:37.727 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:37.727 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:37.727 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:37.727 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:37.730 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.730 INFO:teuthology.orchestra.run.smithi076.stdout:4605 writesame 0x82f1e61 thru 0x82fb9e0 data_size 0x20(0x9b80 bytes) 2024-09-16T13:29:37.734 INFO:teuthology.orchestra.run.smithi076.stdout:4606 write 0x20d046 thru 0x21cd4b (0xfd06 bytes) 2024-09-16T13:29:37.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:37.744 INFO:teuthology.orchestra.run.smithi076.stdout:4607 punch from 0x49a82eb to 0x49adf83, (0x5c98 bytes) 2024-09-16T13:29:37.744 INFO:teuthology.orchestra.run.smithi076.stdout:4608 punch from 0x616f8b5 to 0x617ae4e, (0xb599 bytes) 2024-09-16T13:29:37.744 INFO:teuthology.orchestra.run.smithi076.stdout:4609 write 0x2d3d48 thru 0x2e362e (0xf8e7 bytes) 2024-09-16T13:29:37.753 INFO:teuthology.orchestra.run.smithi076.stdout:4610 write 0x8b2de3c thru 0x8b2e0de (0x2a3 bytes) 2024-09-16T13:29:37.755 INFO:teuthology.orchestra.run.smithi076.stdout:4611 punch from 0xb59580d to 0xb598154, (0x2947 bytes) 2024-09-16T13:29:37.756 INFO:teuthology.orchestra.run.smithi076.stdout:4612 punch from 0x15dab0 to 0x168a56, (0xafa6 bytes) 2024-09-16T13:29:37.763 INFO:teuthology.orchestra.run.smithi076.stdout:4613 trunc from 0xc02c132 to 0x5af008e 2024-09-16T13:29:37.765 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:37.763+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:37.777 INFO:teuthology.orchestra.run.smithi076.stdout:4614 write 0x32ccaaa thru 0x32d4377 (0x78ce bytes) 2024-09-16T13:29:37.781 INFO:teuthology.orchestra.run.smithi076.stdout:4615 read 0x337d0a9 thru 0x3382950 (0x58a8 bytes) 2024-09-16T13:29:37.782 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.796 INFO:teuthology.orchestra.run.smithi076.stdout:4620 write 0xd024b1c thru 0xd031972 (0xce57 bytes) 2024-09-16T13:29:37.804 INFO:teuthology.orchestra.run.smithi076.stdout:4621 read 0x3917eb8 thru 0x391cdc3 (0x4f0c bytes) 2024-09-16T13:29:37.804 INFO:teuthology.orchestra.run.smithi076.stdout:4622 read 0x96df2e5 thru 0x96ebb11 (0xc82d bytes) 2024-09-16T13:29:37.804 INFO:teuthology.orchestra.run.smithi076.stdout:4623 write 0x7c4be23 thru 0x7c500f6 (0x42d4 bytes) 2024-09-16T13:29:37.809 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.810 INFO:teuthology.orchestra.run.smithi076.stdout:4625 read 0x8cf3383 thru 0x8cfa497 (0x7115 bytes) 2024-09-16T13:29:37.810 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:37.810 INFO:teuthology.orchestra.run.smithi076.stdout:4627 write 0x6245f3a thru 0x6253b09 (0xdbd0 bytes) 2024-09-16T13:29:37.818 INFO:teuthology.orchestra.run.smithi076.stdout:4628 writesame 0x8ee2a64 thru 0x8eea5c6 data_size 0x1(0x7b63 bytes) 2024-09-16T13:29:37.858 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.689746327999998, 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.859 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:37.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.684270763000001, 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.865 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:37.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:37.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:37.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.923943956000002, 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:37.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:37.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:37.877 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.907405830999998, 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:37.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.881 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564742381000002, 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:37.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:37.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564720035999997, 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:37.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564702056000002, 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564686252000001, 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:37.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564670691000003, 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.898 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:37.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.902 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564652977000001, 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:37.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:37.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.564626394000001, 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.907 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 58.723072481000003, 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.912 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:37.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:37.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:37.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.918 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 55.525632909999999, 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:37.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:37.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:37.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.950756304000002, 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:37.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.603466346000001, 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.929 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:29:37.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:29:37.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.603186449999999, 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.935 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:37.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:29:37.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:37.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:29:37.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:29:37.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.171494733999999, 2024-09-16T13:29:37.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:37.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:29:37.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:29:37.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:29:37.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.577423352, 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:29:37.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.508693651, 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:29:37.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:29:37.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.497828911999999, 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:37.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:37.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:37.962 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:37.963 INFO:teuthology.orchestra.run.smithi076.stdout:4629 read 0x27c60a7 thru 0x27c6b9a (0xaf4 bytes) 2024-09-16T13:29:37.963 INFO:teuthology.orchestra.run.smithi076.stdout:4630 read 0x54840ce thru 0x548895e (0x4891 bytes) 2024-09-16T13:29:37.964 INFO:teuthology.orchestra.run.smithi076.stdout:4633 clone 39 order 23 su 65536 sc 11 2024-09-16T13:29:37.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:37.965 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:37.962+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:38.065 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:38.063+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:38.103 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:38.103 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:38.103 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:38.103 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:38.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:38.196 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:38.194+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:38.251 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:38.251 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:38.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:38.302 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:38.300+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:38.368 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:38.366+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:38.371 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:29:38.371 INFO:teuthology.orchestra.run.smithi076.stderr:set 4.d pg_upmap mapping to [5,0] 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.708872235999998, 2024-09-16T13:29:38.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:38.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:38.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.687808328000003, 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:38.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.780838068000001, 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:38.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:38.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:38.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72812 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.627310865, 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038465622999999997, 2024-09-16T13:29:38.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72812 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784076+0000", 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784077+0000", 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784078+0000", 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6080000000000001e-06 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784083+0000", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784087+0000", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.818991+0000", 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034903898000000003 2024-09-16T13:29:38.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.819060+0000", 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9875000000000004e-05 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822542+0000", 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034812910000000001 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72813 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.627296953, 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864414, 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72813 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:38.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784092+0000", 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784098+0000", 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5650000000000001e-06 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784102+0000", 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0219999999999998e-06 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822570+0000", 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038468825999999998 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822652+0000", 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822734+0000", 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2639999999999995e-05 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72818 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 540672~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.625461210999999, 2024-09-16T13:29:38.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038576821999999997, 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72818 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:38.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785928+0000", 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8300000000000001e-06 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785934+0000", 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6450000000000002e-06 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785937+0000", 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.823577+0000", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037640060000000003 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824423+0000", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084574599999999998 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824503+0000", 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9583e-05 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.447 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72860 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624920328, 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038621214000000001, 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.448 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72860 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:38.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-07 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786469+0000", 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786475+0000", 2024-09-16T13:29:38.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7420000000000003e-06 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786478+0000", 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.613e-06 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824854+0000", 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038375240999999997 2024-09-16T13:29:38.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824977+0000", 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012308999999999999 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825088+0000", 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111224 2024-09-16T13:29:38.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624904963000001, 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:38.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624889094, 2024-09-16T13:29:38.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:38.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:38.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624874057, 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.464 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:38.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:38.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624859389999999, 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:38.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:38.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624844649, 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:38.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.481 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.62482683, 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:38.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:38.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624642422999999, 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:38.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:38.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624624326999999, 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:38.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624603480999999, 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:38.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:38.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624588685999999, 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.501 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:38.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:38.506 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624574079, 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.511 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.624559623, 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:38.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:38.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:29:38.516 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.128669187, 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:29:38.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:38.521 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:38.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:38.523 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone38 from 0xd031973 (overlap 0x1985f13) to 0x0 2024-09-16T13:29:38.524 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:38.521+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:38.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:38.655 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:38.653+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:38.663 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:38.663 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:38.663 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:38.664 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:38.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:38.727 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:38.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:38.728 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:38.726+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:38.743 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:38.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:38.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:38.759 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:38.774 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:38.774 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:38.790 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:38.790 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:38.805 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:38.806 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:38.813 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:38.813 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:38.821 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:38.821 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:38.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:38.837 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:38.837 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:38.838 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:38.836+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:38.926 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:38.924+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:38.934 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:38.934 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:38.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:39.026 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:39.025+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:39.081 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.081 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:39.128 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:39.126+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:39.227 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.227 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.228 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:39.226+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:39.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:39.329 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:39.327+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:39.377 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.377 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:39.431 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:39.429+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:39.531 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:39.529+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:39.632 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.633 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.633 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:39.631+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:39.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:39.733 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:39.732+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:39.738 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.738 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:39.834 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:39.833+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:39.844 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.844 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:39.936 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:39.934+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:39.952 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:39.952 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:39.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:40.036 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:40.035+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:40.059 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.059 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:40.152 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:40.150+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:40.206 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.207 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:40.257 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:40.255+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:40.339 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:40.337+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:40.354 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.354 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:40.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:29:40.439 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:40.438+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:40.462 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.463 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.471 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:40.540 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:40.539+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:40.609 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.609 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:40.661 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:40.659+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:40.717 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.717 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:40.741 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:40.740+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:40.842 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:40.841+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:40.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:40.860 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:40.860 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:40.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:40.944 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:40.941+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:40.966 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:40.967 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:40.982 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:40.982 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:40.993 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:40.993 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:40.993 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:40.993 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:40.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:40.997 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:41.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:41.012 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:41.029 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:41.044 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:41.042+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:41.045 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:41.059 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:41.075 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:41.075 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:41.103 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:41.103 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:41.103 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:41.103 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:41.103 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:41.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:41.146 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:41.144+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:41.247 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:41.244+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:41.265 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:41.265 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:41.266 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.096118888000007, 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:41.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.268 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:41.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:41.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:41.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:41.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:41.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:41.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:41.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:41.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.090643322999995, 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.279 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:41.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:41.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.330316516000003, 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:41.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:41.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:41.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.313778391, 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:41.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8158 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.295 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971114941000003, 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043510563000000002, 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.296 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8158 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290669+0000", 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290670+0000", 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.65e-07 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290676+0000", 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2160000000000003e-06 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290684+0000", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0569999999999993e-06 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290688+0000", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333830+0000", 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043142126000000003 2024-09-16T13:29:41.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.333897+0000", 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7701999999999997e-05 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334180+0000", 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028229600000000002 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8159 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971092595999998, 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044065944000000003, 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8159 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290692+0000", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290694+0000", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290701+0000", 2024-09-16T13:29:41.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290705+0000", 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0119999999999999e-06 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334224+0000", 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043518814000000003 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334308+0000", 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3407999999999998e-05 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334757+0000", 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00044995899999999999 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.305 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971074616000003, 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:41.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:41.310 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971058812000003, 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:41.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:41.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971043250999998, 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:41.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.319 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.971025537000003, 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:41.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.324 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.970998954000002, 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.325 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:41.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:41.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:41.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 62.129445040999997, 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 58.93200547, 2024-09-16T13:29:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.357128864000003, 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.345 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:41.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.009838905999999, 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:29:41.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:29:41.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:29:41.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.009559009999997, 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.356 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:41.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:29:41.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:41.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:41.362 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.577867294000001, 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:29:41.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:29:41.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.983795912, 2024-09-16T13:29:41.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:29:41.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.915066210999999, 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:41.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:29:41.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.904201472, 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:29:41.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:29:41.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.384 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:41.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:41.386 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-09-16T13:29:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:29:41.387 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:41.383+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:41.486 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:41.484+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:41.551 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:41.551 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:41.551 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:41.551 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:41.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:41.589 INFO:teuthology.orchestra.run.smithi076.stdout:4635 punch from 0x9851515 to 0x9859768, (0x8253 bytes) 2024-09-16T13:29:41.590 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:41.588+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:41.594 INFO:teuthology.orchestra.run.smithi076.stdout:4636 write 0x6cbd4c6 thru 0x6cbe5de (0x1119 bytes) 2024-09-16T13:29:41.599 INFO:teuthology.orchestra.run.smithi076.stdout:4637 writesame 0x60b3aec thru 0x60b4c48 data_size 0x7(0x115d bytes) 2024-09-16T13:29:41.610 INFO:teuthology.orchestra.run.smithi076.stdout:4638 trunc from 0xd031973 to 0x8123f41 2024-09-16T13:29:41.634 INFO:teuthology.orchestra.run.smithi076.stdout:4640 read 0x28edb62 thru 0x28f510c (0x75ab bytes) 2024-09-16T13:29:41.634 INFO:teuthology.orchestra.run.smithi076.stdout:4641 punch from 0x743d548 to 0x74429f3, (0x54ab bytes) 2024-09-16T13:29:41.634 INFO:teuthology.orchestra.run.smithi076.stdout:4642 read 0x376bf62 thru 0x3777fb6 (0xc055 bytes) 2024-09-16T13:29:41.634 INFO:teuthology.orchestra.run.smithi076.stdout:4643 write 0x17ac396 thru 0x17bc067 (0xfcd2 bytes) 2024-09-16T13:29:41.670 INFO:teuthology.orchestra.run.smithi076.stdout:4645 read 0x15c0074 thru 0x15c297d (0x290a bytes) 2024-09-16T13:29:41.674 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:41.674 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:41.674 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:41.682 INFO:teuthology.orchestra.run.smithi076.stdout:4646 writesame 0xceb98f1 thru 0xcebb427 data_size 0x1(0x1b37 bytes) 2024-09-16T13:29:41.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:41.693 INFO:teuthology.orchestra.run.smithi076.stdout:4648 trunc from 0xcebb428 to 0x4126d7d 2024-09-16T13:29:41.694 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:41.692+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:41.781 INFO:teuthology.orchestra.run.smithi076.stdout:4649 trunc from 0x4126d7d to 0x46b837a 2024-09-16T13:29:41.786 INFO:teuthology.orchestra.run.smithi076.stdout:4650 write 0x28c2607 thru 0x28d2472 (0xfe6c bytes) 2024-09-16T13:29:41.787 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:41.785+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:41.798 INFO:teuthology.orchestra.run.smithi076.stdout:4651 writesame 0x61a06ef thru 0x61ae054 data_size 0x1(0xd966 bytes) 2024-09-16T13:29:41.798 INFO:teuthology.orchestra.run.smithi076.stdout:4653 punch from 0x114d63a to 0x115bbb4, (0xe57a bytes) 2024-09-16T13:29:41.798 INFO:teuthology.orchestra.run.smithi076.stdout:4654 punch from 0x2c07247 to 0x2c0e690, (0x7449 bytes) 2024-09-16T13:29:41.798 INFO:teuthology.orchestra.run.smithi076.stdout:4655 write 0x24e9039 thru 0x24f7dcd (0xed95 bytes) 2024-09-16T13:29:41.800 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.801 INFO:teuthology.orchestra.run.smithi076.stdout:4660 writesame 0x5ea0ce8 thru 0x5ea7028 data_size 0x1(0x6341 bytes) 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.124728488000002, 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:41.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:41.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:41.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:41.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:41.836 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.10366458, 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:41.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:41.840 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 55.196694319999999, 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:41.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:41.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:41.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72813 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 286720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.043153204999999, 2024-09-16T13:29:41.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864414, 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72813 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784090+0000", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784092+0000", 2024-09-16T13:29:41.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784098+0000", 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5650000000000001e-06 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.784102+0000", 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0219999999999998e-06 2024-09-16T13:29:41.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822570+0000", 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038468825999999998 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822652+0000", 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.822734+0000", 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2639999999999995e-05 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72818 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 540672~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.041317462999999, 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038576821999999997, 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72818 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:41.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785926+0000", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785928+0000", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8300000000000001e-06 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785934+0000", 2024-09-16T13:29:41.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6450000000000002e-06 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.785937+0000", 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6030000000000001e-06 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.823577+0000", 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037640060000000003 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824423+0000", 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084574599999999998 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824503+0000", 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9583e-05 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72860 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040776579999999, 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038621214000000001, 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72860 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786467+0000", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-07 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786469+0000", 2024-09-16T13:29:41.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786475+0000", 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7420000000000003e-06 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786478+0000", 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.613e-06 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824854+0000", 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038375240999999997 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.824977+0000", 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012308999999999999 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825088+0000", 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111224 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040761215, 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040745346000001, 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:41.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:41.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:41.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040730309000001, 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.869 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040715641999999, 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:41.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:41.877 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040700901000001, 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.881 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040683082000001, 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:41.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.886 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040498674999998, 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040480579, 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:41.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.895 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040459732999999, 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:41.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040444938, 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:41.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040430331, 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.905 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:41.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:41.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.040415875000001, 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:41.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.914 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.544525439000001, 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:41.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:29:41.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:29:41.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.095255429000000003, 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:29:41.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:29:41.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:29:41.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:29:41.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:41.925 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:41.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:4661 trunc from 0x61ae055 to 0x1d476a1 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:4665 trunc from 0x1d476a1 to 0x86fc077 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:4667 trunc from 0x86fc077 to 0x4819fa3 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:4668 read 0x26c764f thru 0x26cf9c1 (0x8373 bytes) 2024-09-16T13:29:41.928 INFO:teuthology.orchestra.run.smithi076.stdout:4670 write 0x6ffd43b thru 0x7003c8c (0x6852 bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4671 read 0x34255ef thru 0x342682f (0x1241 bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4674 write 0x1e9ccc7 thru 0x1e9e72e (0x1a68 bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4675 read 0x48a3560 thru 0x48ab230 (0x7cd1 bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4676 write 0x514e0c2 thru 0x5151cec (0x3c2b bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4678 writesame 0x7340845 thru 0x7347ac2 data_size 0xf(0x727e bytes) 2024-09-16T13:29:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:4680 trunc from 0x7347ac3 to 0x98ee1b5 2024-09-16T13:29:41.929 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:41.924+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:41.934 INFO:teuthology.orchestra.run.smithi076.stdout:4682 write 0xc747f94 thru 0xc753e84 (0xbef1 bytes) 2024-09-16T13:29:41.940 INFO:teuthology.orchestra.run.smithi076.stdout:4684 write 0xbaa613e thru 0xbaaacbe (0x4b81 bytes) 2024-09-16T13:29:41.943 INFO:teuthology.orchestra.run.smithi076.stdout:4688 punch from 0x554b717 to 0x554edc5, (0x36ae bytes) 2024-09-16T13:29:41.944 INFO:teuthology.orchestra.run.smithi076.stdout:4690 writesame 0x150c53f thru 0x151771b data_size 0x1(0xb1dd bytes) 2024-09-16T13:29:41.974 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.974 INFO:teuthology.orchestra.run.smithi076.stdout:4693 write 0x869d902 thru 0x86aa003 (0xc702 bytes) 2024-09-16T13:29:41.978 INFO:teuthology.orchestra.run.smithi076.stdout:4694 read 0xc4ae78c thru 0xc4b4406 (0x5c7b bytes) 2024-09-16T13:29:41.982 INFO:teuthology.orchestra.run.smithi076.stdout:4695 write 0xc971b57 thru 0xc9753ec (0x3896 bytes) 2024-09-16T13:29:41.985 INFO:teuthology.orchestra.run.smithi076.stdout:4696 punch from 0x9d5094 to 0x9d638d, (0x12f9 bytes) 2024-09-16T13:29:41.985 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:41.985 INFO:teuthology.orchestra.run.smithi076.stdout:4698 write 0x5ad9b28 thru 0x5ae34ff (0x99d8 bytes) 2024-09-16T13:29:41.991 INFO:teuthology.orchestra.run.smithi076.stdout:4702 write 0x770a546 thru 0x770bda1 (0x185c bytes) 2024-09-16T13:29:41.992 INFO:teuthology.orchestra.run.smithi076.stdout:4703 read 0xf91073 thru 0xf9176a (0x6f8 bytes) 2024-09-16T13:29:41.993 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.001 INFO:teuthology.orchestra.run.smithi076.stdout:4706 writesame 0xe6b0e4a thru 0xe6bb4ac data_size 0x7(0xa663 bytes) 2024-09-16T13:29:42.001 INFO:teuthology.orchestra.run.smithi076.stdout:4707 punch from 0x248473f to 0x2484898, (0x159 bytes) 2024-09-16T13:29:42.001 INFO:teuthology.orchestra.run.smithi076.stdout:4708 writesame 0x9c8a068 thru 0x9c90f20 data_size 0x1(0x6eb9 bytes) 2024-09-16T13:29:42.021 INFO:teuthology.orchestra.run.smithi076.stdout:4709 writesame 0x5d7eee6 thru 0x5d8d029 data_size 0x1a(0xe144 bytes) 2024-09-16T13:29:42.025 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.026 INFO:teuthology.orchestra.run.smithi076.stdout:4711 writesame 0x4eb16a7 thru 0x4eb9445 data_size 0x1(0x7d9f bytes) 2024-09-16T13:29:42.026 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:42.025+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:42.041 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.041 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:42.041 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:42.041 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.049 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:42.092 INFO:teuthology.orchestra.run.smithi076.stdout:4712 trunc from 0xe6bb4ad to 0x4d7319d 2024-09-16T13:29:42.107 INFO:teuthology.orchestra.run.smithi076.stdout:4714 trunc from 0x4d7319d to 0x6f0d541 2024-09-16T13:29:42.125 INFO:teuthology.orchestra.run.smithi076.stdout:4715 write 0xc426243 thru 0xc429835 (0x35f3 bytes) 2024-09-16T13:29:42.127 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:42.125+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:42.128 INFO:teuthology.orchestra.run.smithi076.stdout:4716 read 0x625678f thru 0x6258135 (0x19a7 bytes) 2024-09-16T13:29:42.128 INFO:teuthology.orchestra.run.smithi076.stdout:4717 write 0x1890a04 thru 0x1892a66 (0x2063 bytes) 2024-09-16T13:29:42.129 INFO:teuthology.orchestra.run.smithi076.stdout:4719 punch from 0x2b83317 to 0x2b928f7, (0xf5e0 bytes) 2024-09-16T13:29:42.136 INFO:teuthology.orchestra.run.smithi076.stdout:4721 write 0xc5ff5df thru 0xc60be1b (0xc83d bytes) 2024-09-16T13:29:42.140 INFO:teuthology.orchestra.run.smithi076.stdout:4722 punch from 0xa890999 to 0xa89fcb4, (0xf31b bytes) 2024-09-16T13:29:42.140 INFO:teuthology.orchestra.run.smithi076.stdout:4724 writesame 0xb147336 thru 0xb15316d data_size 0xc(0xbe38 bytes) 2024-09-16T13:29:42.156 INFO:teuthology.orchestra.run.smithi076.stdout:4731 trunc from 0xc60be1c to 0x2516f78 2024-09-16T13:29:42.168 INFO:teuthology.orchestra.run.smithi076.stdout:4732 read 0x1d6bbf0 thru 0x1d76d1f (0xb130 bytes) 2024-09-16T13:29:42.169 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.169 INFO:teuthology.orchestra.run.smithi076.stdout:4735 write 0x17b1f94 thru 0x17b3664 (0x16d1 bytes) 2024-09-16T13:29:42.189 INFO:teuthology.orchestra.run.smithi076.stdout:4737 writesame 0xd5084f3 thru 0xd50cdd9 data_size 0x1(0x48e7 bytes) 2024-09-16T13:29:42.189 INFO:teuthology.orchestra.run.smithi076.stdout:4739 read 0xc19b201 thru 0xc1a6b8d (0xb98d bytes) 2024-09-16T13:29:42.189 INFO:teuthology.orchestra.run.smithi076.stdout:4740 write 0x9409811 thru 0x9415da8 (0xc598 bytes) 2024-09-16T13:29:42.193 INFO:teuthology.orchestra.run.smithi076.stdout:4741 writesame 0x228c8e6 thru 0x22958f8 data_size 0xb(0x9013 bytes) 2024-09-16T13:29:42.196 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.196 INFO:teuthology.orchestra.run.smithi076.stdout:4744 read 0x5f7420a thru 0x5f7c0ff (0x7ef6 bytes) 2024-09-16T13:29:42.196 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.196 INFO:teuthology.orchestra.run.smithi076.stdout:4747 punch from 0xd3cc27e to 0xd3d982a, (0xd5ac bytes) 2024-09-16T13:29:42.200 INFO:teuthology.orchestra.run.smithi076.stdout:4748 trunc from 0xd50cdda to 0x9f471b6 2024-09-16T13:29:42.208 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:42.208 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.212 INFO:teuthology.orchestra.run.smithi076.stdout:4749 write 0xa7a2554 thru 0xa7afe05 (0xd8b2 bytes) 2024-09-16T13:29:42.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:42.223 INFO:teuthology.orchestra.run.smithi076.stdout:4751 writesame 0xc114e3a thru 0xc1209ab data_size 0x1(0xbb72 bytes) 2024-09-16T13:29:42.223 INFO:teuthology.orchestra.run.smithi076.stdout:4753 write 0xbb2c3fa thru 0xbb32b26 (0x672d bytes) 2024-09-16T13:29:42.228 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:42.226+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:42.229 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.229 INFO:teuthology.orchestra.run.smithi076.stdout:4755 write 0x9629b7d thru 0x9633674 (0x9af8 bytes) 2024-09-16T13:29:42.233 INFO:teuthology.orchestra.run.smithi076.stdout:4756 punch from 0x9a54b23 to 0x9a62746, (0xdc23 bytes) 2024-09-16T13:29:42.233 INFO:teuthology.orchestra.run.smithi076.stdout:4757 writesame 0x395f7d0 thru 0x396d539 data_size 0x5e(0xdd6a bytes) 2024-09-16T13:29:42.235 INFO:teuthology.orchestra.run.smithi076.stdout:4759 write 0x8a10f16 thru 0x8a11a08 (0xaf3 bytes) 2024-09-16T13:29:42.236 INFO:teuthology.orchestra.run.smithi076.stdout:4760 write 0xac3df0b thru 0xac45f4c (0x8042 bytes) 2024-09-16T13:29:42.240 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.240 INFO:teuthology.orchestra.run.smithi076.stdout:4762 read 0x1a52f1a thru 0x1a5579a (0x2881 bytes) 2024-09-16T13:29:42.241 INFO:teuthology.orchestra.run.smithi076.stdout:4763 punch from 0xa6eb01d to 0xa6f8ece, (0xdeb1 bytes) 2024-09-16T13:29:42.241 INFO:teuthology.orchestra.run.smithi076.stdout:4764 write 0x2adb59f thru 0x2adb830 (0x292 bytes) 2024-09-16T13:29:42.242 INFO:teuthology.orchestra.run.smithi076.stdout:4765 write 0x4813525 thru 0x48165d2 (0x30ae bytes) 2024-09-16T13:29:42.244 INFO:teuthology.orchestra.run.smithi076.stdout:4766 write 0xb052821 thru 0xb05ff9e (0xd77e bytes) 2024-09-16T13:29:42.248 INFO:teuthology.orchestra.run.smithi076.stdout:4768 writesame 0x5d7f402 thru 0x5d80fc5 data_size 0x2(0x1bc4 bytes) 2024-09-16T13:29:42.252 INFO:teuthology.orchestra.run.smithi076.stdout:4769 read 0x576e613 thru 0x5772202 (0x3bf0 bytes) 2024-09-16T13:29:42.256 INFO:teuthology.orchestra.run.smithi076.stdout:4770 trunc from 0xc1209ac to 0x9074e54 2024-09-16T13:29:42.277 INFO:teuthology.orchestra.run.smithi076.stdout:4771 writesame 0xbd87c9e thru 0xbd8b23e data_size 0x1(0x35a1 bytes) 2024-09-16T13:29:42.277 INFO:teuthology.orchestra.run.smithi076.stdout:4772 clone 40 order 25 su 2097152 sc 2 2024-09-16T13:29:42.329 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.329 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:42.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.194494899999995, 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:42.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:42.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.335 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.803676985999999, 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:42.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:42.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.166002517000003, 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.085951311999999, 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:42.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.085923991999998, 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.354 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:42.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:42.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:42.358 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.010970479999997, 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:42.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:42.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:42.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.005675203000003, 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:42.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:42.368 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.005116645000001, 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:42.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:42.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53295 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.332540297000001, 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091736779999999993, 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53295 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993673+0000", 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993674+0000", 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3099999999999997e-07 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993676+0000", 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993684+0000", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9710000000000004e-06 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993689+0000", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8099999999999997e-06 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002744+0000", 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090548969999999993 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002794+0000", 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0244999999999998e-05 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002847+0000", 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2241999999999999e-05 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.378 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53296 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.332491355999998, 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093249630000000003, 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53296 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:42.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993725+0000", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993734+0000", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993738+0000", 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1230000000000002e-06 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002875+0000", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091369250000000006 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002952+0000", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7404999999999995e-05 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003047+0000", 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4820000000000004e-05 2024-09-16T13:29:42.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.332463142000002, 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:42.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.332436897000001, 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:42.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:42.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.332416988999999, 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:42.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.396 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.755938986, 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.590591348, 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.403 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:42.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:42.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.574398602999999, 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:42.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.4476807019999995, 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:29:42.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:29:42.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.3738505629999995, 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:29:42.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:42.423 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2551093269999996, 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:29:42.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:29:42.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 8.2429376380000008, 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:29:42.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:29:42.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.432 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.433 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:42.327+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:42.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:42.434 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:42.431+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:42.456 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone39 from 0xbd8b23f (overlap 0x1d476a1) to 0x0 2024-09-16T13:29:42.519 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-09-16T13:29:42.521 INFO:teuthology.orchestra.run.smithi076.stdout:4773 read 0x133377b thru 0x133b4b0 (0x7d36 bytes) 2024-09-16T13:29:42.522 INFO:teuthology.orchestra.run.smithi076.stdout:4774 read 0x20a89ed thru 0x20b8903 (0xff17 bytes) 2024-09-16T13:29:42.522 INFO:teuthology.orchestra.run.smithi076.stdout:4775 read 0x6609d14 thru 0x6618be2 (0xeecf bytes) 2024-09-16T13:29:42.523 INFO:teuthology.orchestra.run.smithi076.stdout:4776 read 0x31ccacc thru 0x31cdef7 (0x142c bytes) 2024-09-16T13:29:42.524 INFO:teuthology.orchestra.run.smithi076.stdout:4778 write 0x5f23120 thru 0x5f2c135 (0x9016 bytes) 2024-09-16T13:29:42.534 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:42.532+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:42.539 INFO:teuthology.orchestra.run.smithi076.stdout:4779 write 0x7b46a54 thru 0x7b564c6 (0xfa73 bytes) 2024-09-16T13:29:42.551 INFO:teuthology.orchestra.run.smithi076.stdout:4780 trunc from 0xbd8b23f to 0x4a08793 2024-09-16T13:29:42.566 INFO:teuthology.orchestra.run.smithi076.stdout:4781 punch from 0x14004ba to 0x14050bb, (0x4c01 bytes) 2024-09-16T13:29:42.567 INFO:teuthology.orchestra.run.smithi076.stdout:4782 punch from 0x12402ee to 0x1243b26, (0x3838 bytes) 2024-09-16T13:29:42.567 INFO:teuthology.orchestra.run.smithi076.stdout:4783 read 0x35140ca thru 0x3522b9c (0xead3 bytes) 2024-09-16T13:29:42.567 INFO:teuthology.orchestra.run.smithi076.stdout:4784 read 0x49efdc0 thru 0x49f26c1 (0x2902 bytes) 2024-09-16T13:29:42.570 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.570 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:42.570 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:42.571 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.579 INFO:teuthology.orchestra.run.smithi076.stdout:4785 write 0x929b604 thru 0x92a27e0 (0x71dd bytes) 2024-09-16T13:29:42.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:42.593 INFO:teuthology.orchestra.run.smithi076.stdout:4788 trunc from 0x92a27e1 to 0x5170b7 2024-09-16T13:29:42.634 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:42.633+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:42.675 INFO:teuthology.orchestra.run.smithi076.stdout:4789 trunc from 0x5170b7 to 0xa167b33 2024-09-16T13:29:42.691 INFO:teuthology.orchestra.run.smithi076.stdout:4790 trunc from 0xa167b33 to 0x1211fc6 2024-09-16T13:29:42.697 INFO:teuthology.orchestra.run.smithi076.stdout:4792 punch from 0xe7f0d6 to 0xe8a54e, (0xb478 bytes) 2024-09-16T13:29:42.697 INFO:teuthology.orchestra.run.smithi076.stdout:4793 read 0xa9af2d thru 0xa9c709 (0x17dd bytes) 2024-09-16T13:29:42.703 INFO:teuthology.orchestra.run.smithi076.stdout:4794 write 0x1c1a7ca thru 0x1c29ce5 (0xf51c bytes) 2024-09-16T13:29:42.713 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.713 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:42.713 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:42.714 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:42.714 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.719 INFO:teuthology.orchestra.run.smithi076.stdout:4795 trunc from 0x1c29ce6 to 0xd01e353 2024-09-16T13:29:42.724 INFO:teuthology.orchestra.run.smithi076.stdout:4796 read 0x7646b2f thru 0x765699e (0xfe70 bytes) 2024-09-16T13:29:42.724 INFO:teuthology.orchestra.run.smithi076.stdout:4798 writesame 0x7dc4df6 thru 0x7dc76d3 data_size 0x1(0x28de bytes) 2024-09-16T13:29:42.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:42.744 INFO:teuthology.orchestra.run.smithi076.stdout:4800 write 0x7efbc45 thru 0x7f08f7c (0xd338 bytes) 2024-09-16T13:29:42.744 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:42.742+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:42.749 INFO:teuthology.orchestra.run.smithi076.stdout:4802 write 0x2d4aeb thru 0x2d88de (0x3df4 bytes) 2024-09-16T13:29:42.751 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.751 INFO:teuthology.orchestra.run.smithi076.stdout:4804 read 0x31dd217 thru 0x31e912d (0xbf17 bytes) 2024-09-16T13:29:42.751 INFO:teuthology.orchestra.run.smithi076.stdout:4806 punch from 0x2bd9174 to 0x2be177f, (0x860b bytes) 2024-09-16T13:29:42.751 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.752 INFO:teuthology.orchestra.run.smithi076.stdout:4808 write 0xada6333 thru 0xada7286 (0xf54 bytes) 2024-09-16T13:29:42.754 INFO:teuthology.orchestra.run.smithi076.stdout:4809 punch from 0x2e39165 to 0x2e464f4, (0xd38f bytes) 2024-09-16T13:29:42.754 INFO:teuthology.orchestra.run.smithi076.stdout:4811 writesame 0x4edfb0f thru 0x4ee9703 data_size 0x5(0x9bf5 bytes) 2024-09-16T13:29:42.771 INFO:teuthology.orchestra.run.smithi076.stdout:4812 write 0x6df42e1 thru 0x6e02cae (0xe9ce bytes) 2024-09-16T13:29:42.774 INFO:teuthology.orchestra.run.smithi076.stdout:4813 read 0x609f628 thru 0x60a9f21 (0xa8fa bytes) 2024-09-16T13:29:42.784 INFO:teuthology.orchestra.run.smithi076.stdout:4814 trunc from 0xd01e353 to 0x4f226ca 2024-09-16T13:29:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:4815 punch from 0x41306b9 to 0x41404b9, (0xfe00 bytes) 2024-09-16T13:29:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:4818 read 0x1815fc2 thru 0x181c328 (0x6367 bytes) 2024-09-16T13:29:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:4819 read 0x1d6e307 thru 0x1d73b6c (0x5866 bytes) 2024-09-16T13:29:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:4820 write 0x23b04d9 thru 0x23bb0f5 (0xac1d bytes) 2024-09-16T13:29:42.798 INFO:teuthology.orchestra.run.smithi076.stdout:4821 write 0x1e9c138 thru 0x1ea9b22 (0xd9eb bytes) 2024-09-16T13:29:42.802 INFO:teuthology.orchestra.run.smithi076.stdout:4822 write 0x362d2ca thru 0x363ab01 (0xd838 bytes) 2024-09-16T13:29:42.806 INFO:teuthology.orchestra.run.smithi076.stdout:4824 punch from 0x1ac5b0e to 0x1acf9ab, (0x9e9d bytes) 2024-09-16T13:29:42.806 INFO:teuthology.orchestra.run.smithi076.stdout:4826 writesame 0x333aa29 thru 0x33408ae data_size 0x6d(0x5e86 bytes) 2024-09-16T13:29:42.808 INFO:teuthology.orchestra.run.smithi076.stdout:4829 read 0xcdfe5b thru 0xce4576 (0x471c bytes) 2024-09-16T13:29:42.808 INFO:teuthology.orchestra.run.smithi076.stdout:4830 read 0x45d2c58 thru 0x45de71c (0xbac5 bytes) 2024-09-16T13:29:42.817 INFO:teuthology.orchestra.run.smithi076.stdout:4831 write 0x7fa54a8 thru 0x7fb5110 (0xfc69 bytes) 2024-09-16T13:29:42.818 INFO:teuthology.orchestra.run.smithi076.stdout:4832 writesame 0x76a7e36 thru 0x76b1657 data_size 0x3(0x9822 bytes) 2024-09-16T13:29:42.836 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:42.836 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:42.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:42.836 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:42.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.837 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.837 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:42.837 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589859892, 2024-09-16T13:29:42.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:29:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:42.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.839 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:42.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:42.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:42.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:42.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:42.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:29:42.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:29:42.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:29:42.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:29:42.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589797419, 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:42.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:29:42.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:29:42.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589752765999997, 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.859 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:29:42.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:42.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589679664999998, 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:42.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:42.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:29:42.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:29:42.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589661159000002, 2024-09-16T13:29:42.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:42.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:42.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589487206999998, 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:29:42.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:42.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:29:42.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.883 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.589448652999998, 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:42.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:29:42.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:29:42.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:29:42.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:29:42.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.588489959, 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:29:42.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:42.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.894 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:42.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:42.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:42.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:42.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:42.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:29:42.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.588432161999997, 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.901 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:29:42.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:29:42.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.588338286000003, 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.907 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:42.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:42.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.588310297, 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.913 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:42.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:29:42.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.587036271000002, 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:42.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.924 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.587013024999997, 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:42.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.930 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.859666875999999, 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:29:42.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:29:42.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:71847 4.19 4:9aaa0942:::rbd_data.11a911a8a02a.0000000000000035:1d [read 770048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.0561565, 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059204723000000001, 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.937 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 71847 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775952+0000", 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9999999999999999e-07 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775954+0000", 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5090000000000001e-06 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775959+0000", 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2209999999999996e-06 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.775962+0000", 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0910000000000001e-06 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.834734+0000", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058771692 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.834803+0000", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9303999999999999e-05 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.835157+0000", 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035360600000000002 2024-09-16T13:29:42.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:71990 4.11 4:8febe650:::rbd_data.11a911a8a02a.0000000000000005:1d [read 860160~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 19.051303327999999, 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059142841000000002, 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 71990 2024-09-16T13:29:42.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780805+0000", 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780806+0000", 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7800000000000002e-07 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780807+0000", 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.945e-06 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780813+0000", 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4229999999999996e-06 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.780816+0000", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-06 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839804+0000", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058987824000000001 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839870+0000", 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5605999999999995e-05 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.839948+0000", 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8145000000000005e-05 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.555038973999999, 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:29:42.946 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:29:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:29:42.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.542318143999999, 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:29:42.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:29:42.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.319284920000001, 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:42.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:29:42.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.839711842, 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:42.962 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:42.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:42.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:29:42.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:29:42.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:42.970 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:42.971 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.972 INFO:teuthology.orchestra.run.smithi076.stdout:4834 trunc from 0x7fb5111 to 0xc451832 2024-09-16T13:29:42.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:42.975 INFO:teuthology.orchestra.run.smithi076.stdout:4835 punch from 0x9a6edda to 0x9a7c679, (0xd89f bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4836 trunc from 0xc451832 to 0xd440385 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4839 read 0x44239e3 thru 0x4427dda (0x43f8 bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4841 write 0x6ccaeec thru 0x6cd457d (0x9692 bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4842 punch from 0x9da7575 to 0x9dab305, (0x3d90 bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4843 write 0xc72daee thru 0xc7310c3 (0x35d6 bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4846 trunc from 0xd440385 to 0x56c67d7 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4849 read 0x32283d0 thru 0x322f4ab (0x70dc bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4850 read 0x372f430 thru 0x3734ccc (0x589d bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4852 write 0xbc5a185 thru 0xbc673e1 (0xd25d bytes) 2024-09-16T13:29:42.976 INFO:teuthology.orchestra.run.smithi076.stdout:4854 write 0x1a7513b thru 0x1a7575e (0x624 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4855 read 0x5398629 thru 0x53a2407 (0x9ddf bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4856 read 0x61b2191 thru 0x61bc638 (0xa4a8 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4857 punch from 0x212149 to 0x2146df, (0x2596 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4858 read 0x6c488a0 thru 0x6c48c51 (0x3b2 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4859 read 0x39d6ccf thru 0x39e2c01 (0xbf33 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4860 punch from 0x1c8c8a1 to 0x1c8dcc0, (0x141f bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4861 read 0x3775274 thru 0x3784503 (0xf290 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4864 write 0xba6eaa5 thru 0xba7d8c9 (0xee25 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4866 writesame 0x1c197c8 thru 0x1c1e5b0 data_size 0x1(0x4de9 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4867 trunc from 0xbc673e2 to 0x6b98af9 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4870 punch from 0xcc7d2c to 0xcc80c4, (0x398 bytes) 2024-09-16T13:29:42.977 INFO:teuthology.orchestra.run.smithi076.stdout:4872 read 0x541c237 thru 0x541d954 (0x171e bytes) 2024-09-16T13:29:42.978 INFO:teuthology.orchestra.run.smithi076.stdout:4874 trunc from 0x6b98af9 to 0x6649e2c 2024-09-16T13:29:42.978 INFO:teuthology.orchestra.run.smithi076.stdout:4877 read 0x546d5e8 thru 0x547d560 (0xff79 bytes) 2024-09-16T13:29:42.978 INFO:teuthology.orchestra.run.smithi076.stdout:4878 write 0x7ed6ce7 thru 0x7ee3fb7 (0xd2d1 bytes) 2024-09-16T13:29:42.978 INFO:teuthology.orchestra.run.smithi076.stdout:4879 trunc from 0x7ee3fb8 to 0xae1b32d 2024-09-16T13:29:42.978 INFO:teuthology.orchestra.run.smithi076.stdout:4880 writesame 0xdd3e62 thru 0xddbc34 data_size 0x9(0x7dd3 bytes) 2024-09-16T13:29:42.978 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:42.970+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:42.979 INFO:teuthology.orchestra.run.smithi076.stdout:4881 writesame 0x1d58888 thru 0x1d5d67d data_size 0x11(0x4df6 bytes) 2024-09-16T13:29:42.979 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:42.984 INFO:teuthology.orchestra.run.smithi076.stdout:4883 trunc from 0xae1b32d to 0x69b1c0d 2024-09-16T13:29:42.991 INFO:teuthology.orchestra.run.smithi076.stdout:4884 write 0x5280b4a thru 0x5284b5d (0x4014 bytes) 2024-09-16T13:29:42.992 INFO:teuthology.orchestra.run.smithi076.stdout:4885 punch from 0xbc9309 to 0xbd3cb0, (0xa9a7 bytes) 2024-09-16T13:29:43.002 INFO:teuthology.orchestra.run.smithi076.stdout:4886 writesame 0x9f0b7f6 thru 0x9f1922c data_size 0x9(0xda37 bytes) 2024-09-16T13:29:43.002 INFO:teuthology.orchestra.run.smithi076.stdout:4887 read 0x1a5d16 thru 0x1a78b3 (0x1b9e bytes) 2024-09-16T13:29:43.002 INFO:teuthology.orchestra.run.smithi076.stdout:4888 write 0x235965e thru 0x236658c (0xcf2f bytes) 2024-09-16T13:29:43.006 INFO:teuthology.orchestra.run.smithi076.stdout:4889 punch from 0x5b8a44a to 0x5b8f602, (0x51b8 bytes) 2024-09-16T13:29:43.008 INFO:teuthology.orchestra.run.smithi076.stdout:4892 trunc from 0x9f1922d to 0xb795d09 2024-09-16T13:29:43.013 INFO:teuthology.orchestra.run.smithi076.stdout:4893 writesame 0x55ac55c thru 0x55b608f data_size 0x56(0x9b34 bytes) 2024-09-16T13:29:43.022 INFO:teuthology.orchestra.run.smithi076.stdout:4894 write 0xca2bdd9 thru 0xca390bd (0xd2e5 bytes) 2024-09-16T13:29:43.025 INFO:teuthology.orchestra.run.smithi076.stdout:4895 read 0x7bea620 thru 0x7bed240 (0x2c21 bytes) 2024-09-16T13:29:43.025 INFO:teuthology.orchestra.run.smithi076.stdout:4898 writesame 0x84e5ff thru 0x85a8a1 data_size 0x11(0xc2a3 bytes) 2024-09-16T13:29:43.039 INFO:teuthology.orchestra.run.smithi076.stdout:4899 trunc from 0xca390be to 0x457f39a 2024-09-16T13:29:43.069 INFO:teuthology.orchestra.run.smithi076.stdout:4900 write 0xe750874 thru 0xe755ed0 (0x565d bytes) 2024-09-16T13:29:43.072 INFO:teuthology.orchestra.run.smithi076.stdout:4901 read 0x3fbed91 thru 0x3fc4340 (0x55b0 bytes) 2024-09-16T13:29:43.073 INFO:teuthology.orchestra.run.smithi076.stdout:4902 write 0x12fac32 thru 0x1300812 (0x5be1 bytes) 2024-09-16T13:29:43.073 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:43.071+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:43.080 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:43.080 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:43.080 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:43.080 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:43.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:43.088 INFO:teuthology.orchestra.run.smithi076.stdout:4903 trunc from 0xe755ed1 to 0x33b2c2f 2024-09-16T13:29:43.089 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:43.103 INFO:teuthology.orchestra.run.smithi076.stdout:4904 read 0x10e43a6 thru 0x10e6666 (0x22c1 bytes) 2024-09-16T13:29:43.103 INFO:teuthology.orchestra.run.smithi076.stdout:4905 punch from 0x1046ad8 to 0x105369b, (0xcbc3 bytes) 2024-09-16T13:29:43.103 INFO:teuthology.orchestra.run.smithi076.stdout:4906 punch from 0x6d887 to 0x71a8b, (0x4204 bytes) 2024-09-16T13:29:43.103 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.115 INFO:teuthology.orchestra.run.smithi076.stdout:4910 write 0x85c7b5f thru 0x85c8288 (0x72a bytes) 2024-09-16T13:29:43.118 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.131 INFO:teuthology.orchestra.run.smithi076.stdout:4913 writesame 0xd11b650 thru 0xd11fb18 data_size 0x1(0x44c9 bytes) 2024-09-16T13:29:43.131 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.131 INFO:teuthology.orchestra.run.smithi076.stdout:4915 write 0x28ce949 thru 0x28d3a29 (0x50e1 bytes) 2024-09-16T13:29:43.133 INFO:teuthology.orchestra.run.smithi076.stdout:4916 trunc from 0xd11fb19 to 0xd508106 2024-09-16T13:29:43.149 INFO:teuthology.orchestra.run.smithi076.stdout:4917 trunc from 0xd508106 to 0x5053d2e 2024-09-16T13:29:43.159 INFO:teuthology.orchestra.run.smithi076.stdout:4919 read 0x24d1b7f thru 0x24d6eaa (0x532c bytes) 2024-09-16T13:29:43.160 INFO:teuthology.orchestra.run.smithi076.stdout:4920 write 0x2943eb0 thru 0x294c1dd (0x832e bytes) 2024-09-16T13:29:43.177 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:43.172+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:43.177 INFO:teuthology.orchestra.run.smithi076.stdout:4921 writesame 0xb43f8d2 thru 0xb4481fe data_size 0x1(0x892d bytes) 2024-09-16T13:29:43.177 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.185 INFO:teuthology.orchestra.run.smithi076.stdout:4923 trunc from 0xb4481ff to 0x409e379 2024-09-16T13:29:43.190 INFO:teuthology.orchestra.run.smithi076.stdout:4926 read 0xf2ad1b thru 0xf2e0b4 (0x339a bytes) 2024-09-16T13:29:43.191 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:43.191 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:43.191 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:43.191 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:43.191 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:43.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:43.207 INFO:teuthology.orchestra.run.smithi076.stdout:4928 writesame 0xded3dcc thru 0xded6726 data_size 0x1(0x295b bytes) 2024-09-16T13:29:43.208 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.208 INFO:teuthology.orchestra.run.smithi076.stdout:4930 read 0x342330e thru 0x342be7d (0x8b70 bytes) 2024-09-16T13:29:43.208 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.208 INFO:teuthology.orchestra.run.smithi076.stdout:4933 read 0x8734439 thru 0x87442b0 (0xfe78 bytes) 2024-09-16T13:29:43.208 INFO:teuthology.orchestra.run.smithi076.stdout:4934 write 0xa955b0d thru 0xa956d4b (0x123f bytes) 2024-09-16T13:29:43.210 INFO:teuthology.orchestra.run.smithi076.stdout:4937 read 0xc2274d4 thru 0xc230ba8 (0x96d5 bytes) 2024-09-16T13:29:43.211 INFO:teuthology.orchestra.run.smithi076.stdout:4940 write 0x80bf795 thru 0x80c7cfa (0x8566 bytes) 2024-09-16T13:29:43.213 INFO:teuthology.orchestra.run.smithi076.stdout:4942 punch from 0x58df362 to 0x58e634f, (0x6fed bytes) 2024-09-16T13:29:43.213 INFO:teuthology.orchestra.run.smithi076.stdout:4943 write 0xcbff121 thru 0xcc09db5 (0xac95 bytes) 2024-09-16T13:29:43.215 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:43.216 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:43.220 INFO:teuthology.orchestra.run.smithi076.stdout:4944 write 0x16dd8d3 thru 0x16e84b3 (0xabe1 bytes) 2024-09-16T13:29:43.223 INFO:teuthology.orchestra.run.smithi076.stdout:4945 write 0xdbd7160 thru 0xdbe301a (0xbebb bytes) 2024-09-16T13:29:43.226 INFO:teuthology.orchestra.run.smithi076.stdout:4946 read 0x5aeb13c thru 0x5af9ac1 (0xe986 bytes) 2024-09-16T13:29:43.226 INFO:teuthology.orchestra.run.smithi076.stdout:4947 write 0x32600c2 thru 0x326ffb5 (0xfef4 bytes) 2024-09-16T13:29:43.230 INFO:teuthology.orchestra.run.smithi076.stdout:4948 write 0x869e484 thru 0x86aa91b (0xc498 bytes) 2024-09-16T13:29:43.233 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:43.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:43.235 INFO:teuthology.orchestra.run.smithi076.stdout:4949 write 0x7ff3967 thru 0x7ff98f0 (0x5f8a bytes) 2024-09-16T13:29:43.238 INFO:teuthology.orchestra.run.smithi076.stdout:4950 read 0x6ce45c3 thru 0x6cefc6a (0xb6a8 bytes) 2024-09-16T13:29:43.245 INFO:teuthology.orchestra.run.smithi076.stdout:4951 write 0xe0161b0 thru 0xe01aedc (0x4d2d bytes) 2024-09-16T13:29:43.247 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.247 INFO:teuthology.orchestra.run.smithi076.stdout:4953 write 0x855bebf thru 0x855efab (0x30ed bytes) 2024-09-16T13:29:43.248 INFO:teuthology.orchestra.run.smithi076.stdout:4954 punch from 0xad3f4db to 0xad4b815, (0xc33a bytes) 2024-09-16T13:29:43.248 INFO:teuthology.orchestra.run.smithi076.stdout:4955 read 0xc62cc00 thru 0xc63977f (0xcb80 bytes) 2024-09-16T13:29:43.248 INFO:teuthology.orchestra.run.smithi076.stdout:4956 writesame 0x19ff460 thru 0x1a0c43d data_size 0x93(0xcfde bytes) 2024-09-16T13:29:43.250 INFO:teuthology.orchestra.run.smithi076.stdout:4957 read 0x1db8887 thru 0x1dbceac (0x4626 bytes) 2024-09-16T13:29:43.250 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:43.250 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:43.265 INFO:teuthology.orchestra.run.smithi076.stdout:4958 trunc from 0xe01aedd to 0x15e9b3f 2024-09-16T13:29:43.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:43.267 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:43.277 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.277 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:43.276+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:43.281 INFO:teuthology.orchestra.run.smithi076.stdout:4961 writesame 0x1afb843 thru 0x1b02ed3 data_size 0x1(0x7691 bytes) 2024-09-16T13:29:43.283 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:43.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:43.300 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:43.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:43.303 INFO:teuthology.orchestra.run.smithi076.stdout:4963 write 0xe44d9d3 thru 0xe4504ef (0x2b1d bytes) 2024-09-16T13:29:43.306 INFO:teuthology.orchestra.run.smithi076.stdout:4964 writesame 0x7a7fef0 thru 0x7a8c7e2 data_size 0x1(0xc8f3 bytes) 2024-09-16T13:29:43.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:43.317 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:43.332 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:43.332 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:43.343 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:43.343 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:43.352 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:43.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:43.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:43.388 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-16T13:29:43.388 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:29:43.389 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:29:43.389 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-16T13:29:43.396 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:43.394+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:43.397 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.398 INFO:teuthology.orchestra.run.smithi076.stdout:4966 write 0x6054846 thru 0x6062b71 (0xe32c bytes) 2024-09-16T13:29:43.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:29:43.409 INFO:teuthology.orchestra.run.smithi076.stdout:4967 read 0xad84075 thru 0xad8d8f6 (0x9882 bytes) 2024-09-16T13:29:43.409 INFO:teuthology.orchestra.run.smithi076.stdout:4968 write 0xb69b531 thru 0xb69be59 (0x929 bytes) 2024-09-16T13:29:43.412 INFO:teuthology.orchestra.run.smithi076.stdout:4969 write 0xb2ebec thru 0xb3d058 (0xe46d bytes) 2024-09-16T13:29:43.417 INFO:teuthology.orchestra.run.smithi076.stdout:4970 read 0x2849614 thru 0x2857f0a (0xe8f7 bytes) 2024-09-16T13:29:43.418 INFO:teuthology.orchestra.run.smithi076.stdout:4972 write 0x28710c1 thru 0x287270d (0x164d bytes) 2024-09-16T13:29:43.420 INFO:teuthology.orchestra.run.smithi076.stdout:4973 write 0xe2db9af thru 0xe2e9e94 (0xe4e6 bytes) 2024-09-16T13:29:43.426 INFO:teuthology.orchestra.run.smithi076.stdout:4974 read 0xb6bd205 thru 0xb6c9374 (0xc170 bytes) 2024-09-16T13:29:43.426 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.427 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.427 INFO:teuthology.orchestra.run.smithi076.stdout:4978 read 0xccabfb4 thru 0xccb7231 (0xb27e bytes) 2024-09-16T13:29:43.442 INFO:teuthology.orchestra.run.smithi076.stdout:4979 trunc from 0xe4504f0 to 0x13b59ea 2024-09-16T13:29:43.454 INFO:teuthology.orchestra.run.smithi076.stdout:4982 read 0x1148915 thru 0x114d9a5 (0x5091 bytes) 2024-09-16T13:29:43.455 INFO:teuthology.orchestra.run.smithi076.stdout:4983 punch from 0xa89072 to 0xa91297, (0x8225 bytes) 2024-09-16T13:29:43.455 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.459 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:43.459 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:43.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:43.469 INFO:teuthology.orchestra.run.smithi076.stdout:4987 write 0x8186680 thru 0x818dbd9 (0x755a bytes) 2024-09-16T13:29:43.472 INFO:teuthology.orchestra.run.smithi076.stdout:4988 writesame 0x446e40a thru 0x447dbaa data_size 0x71(0xf7a1 bytes) 2024-09-16T13:29:43.475 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.475 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.475 INFO:teuthology.orchestra.run.smithi076.stdout:4993 read 0x4edf1cd thru 0x4eec0f7 (0xcf2b bytes) 2024-09-16T13:29:43.476 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.476 INFO:teuthology.orchestra.run.smithi076.stdout:4995 read 0x45181ee thru 0x4524c44 (0xca57 bytes) 2024-09-16T13:29:43.476 INFO:teuthology.orchestra.run.smithi076.stdout:4996 read 0xc7b382 thru 0xc87835 (0xc4b4 bytes) 2024-09-16T13:29:43.476 INFO:teuthology.orchestra.run.smithi076.stdout:4997 writesame 0x2716fec thru 0x2721b29 data_size 0x17(0xab3e bytes) 2024-09-16T13:29:43.479 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:43.478+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:43.480 INFO:teuthology.orchestra.run.smithi076.stdout:4998 punch from 0x2e020da to 0x2e047ee, (0x2714 bytes) 2024-09-16T13:29:43.480 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.489 INFO:teuthology.orchestra.run.smithi076.stdout:5002 write 0xb98188c thru 0xb9818f7 (0x6c bytes) 2024-09-16T13:29:43.490 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.499 INFO:teuthology.orchestra.run.smithi076.stdout:5004 write 0xe91b7b9 thru 0xe9205e5 (0x4e2d bytes) 2024-09-16T13:29:43.502 INFO:teuthology.orchestra.run.smithi076.stdout:5005 write 0x1c1a979 thru 0x1c26c03 (0xc28b bytes) 2024-09-16T13:29:43.507 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:29:43.507 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:43.508 INFO:teuthology.orchestra.run.smithi076.stdout:5006 read 0x8218ad1 thru 0x8221cc4 (0x91f4 bytes) 2024-09-16T13:29:43.508 INFO:teuthology.orchestra.run.smithi076.stdout:5007 read 0x13b8644 thru 0x13c406f (0xba2c bytes) 2024-09-16T13:29:43.515 INFO:teuthology.orchestra.run.smithi076.stdout:5008 trunc from 0xe9205e6 to 0x9282445 2024-09-16T13:29:43.526 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.526 INFO:teuthology.orchestra.run.smithi076.stdout:5010 read 0x8e16df6 thru 0x8e1c0ae (0x52b9 bytes) 2024-09-16T13:29:43.533 INFO:teuthology.orchestra.run.smithi076.stdout:5012 trunc from 0x9282445 to 0xea38267 2024-09-16T13:29:43.540 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.540 INFO:teuthology.orchestra.run.smithi076.stdout:5016 punch from 0xbc47122 to 0xbc552ed, (0xe1cb bytes) 2024-09-16T13:29:43.540 INFO:teuthology.orchestra.run.smithi076.stdout:5017 write 0x550d205 thru 0x55117bf (0x45bb bytes) 2024-09-16T13:29:43.559 INFO:teuthology.orchestra.run.smithi076.stdout:5018 trunc from 0xea38267 to 0x189575 2024-09-16T13:29:43.575 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.575 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.580 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:43.579+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:43.595 INFO:teuthology.orchestra.run.smithi076.stdout:5021 write 0xb1da124 thru 0xb1e56c8 (0xb5a5 bytes) 2024-09-16T13:29:43.598 INFO:teuthology.orchestra.run.smithi076.stdout:5022 read 0x870ee3e thru 0x8710109 (0x12cc bytes) 2024-09-16T13:29:43.599 INFO:teuthology.orchestra.run.smithi076.stdout:5023 write 0x232f0f4 thru 0x232ffae (0xebb bytes) 2024-09-16T13:29:43.601 INFO:teuthology.orchestra.run.smithi076.stdout:5024 write 0x4f24253 thru 0x4f30960 (0xc70e bytes) 2024-09-16T13:29:43.607 INFO:teuthology.orchestra.run.smithi076.stdout:5025 punch from 0x6f9eb9b to 0x6fa5def, (0x7254 bytes) 2024-09-16T13:29:43.607 INFO:teuthology.orchestra.run.smithi076.stdout:5026 punch from 0x2790190 to 0x2795401, (0x5271 bytes) 2024-09-16T13:29:43.607 INFO:teuthology.orchestra.run.smithi076.stdout:5027 read 0x7c9c4c thru 0x7cd793 (0x3b48 bytes) 2024-09-16T13:29:43.607 INFO:teuthology.orchestra.run.smithi076.stdout:5028 read 0x5ec11c2 thru 0x5ecdf6a (0xcda9 bytes) 2024-09-16T13:29:43.607 INFO:teuthology.orchestra.run.smithi076.stdout:5029 writesame 0x56ee8f0 thru 0x56f2d9e data_size 0x1(0x44af bytes) 2024-09-16T13:29:43.612 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:43.612 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:43.621 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:43.633 INFO:teuthology.orchestra.run.smithi076.stdout:5030 punch from 0x7829cc8 to 0x783976d, (0xfaa5 bytes) 2024-09-16T13:29:43.633 INFO:teuthology.orchestra.run.smithi076.stdout:5032 read 0x413f146 thru 0x413fc70 (0xb2b bytes) 2024-09-16T13:29:43.633 INFO:teuthology.orchestra.run.smithi076.stdout:5033 punch from 0x80da89d to 0x80e9f0c, (0xf66f bytes) 2024-09-16T13:29:43.634 INFO:teuthology.orchestra.run.smithi076.stdout:5038 trunc from 0xb1e56c9 to 0xc3b628c 2024-09-16T13:29:43.646 INFO:teuthology.orchestra.run.smithi076.stdout:5039 write 0xc74da03 thru 0xc7579c1 (0x9fbf bytes) 2024-09-16T13:29:43.649 INFO:teuthology.orchestra.run.smithi076.stdout:5040 write 0x6f55771 thru 0x6f5b5c7 (0x5e57 bytes) 2024-09-16T13:29:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:5043 punch from 0x2b8c38b to 0x2b993bf, (0xd034 bytes) 2024-09-16T13:29:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:5045 read 0x874f0c7 thru 0x8751a05 (0x293f bytes) 2024-09-16T13:29:43.653 INFO:teuthology.orchestra.run.smithi076.stdout:5047 write 0xa1c2ff2 thru 0xa1cb969 (0x8978 bytes) 2024-09-16T13:29:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:5050 read 0x5be8ead thru 0x5bf2e83 (0x9fd7 bytes) 2024-09-16T13:29:43.655 INFO:teuthology.orchestra.run.smithi076.stdout:5051 writesame 0x90f4928 thru 0x90fb3b3 data_size 0x6(0x6a8c bytes) 2024-09-16T13:29:43.663 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.663 INFO:teuthology.orchestra.run.smithi076.stdout:5053 read 0xab9f8b7 thru 0xaba052f (0xc79 bytes) 2024-09-16T13:29:43.663 INFO:teuthology.orchestra.run.smithi076.stdout:5054 read 0x847381a thru 0x848070c (0xcef3 bytes) 2024-09-16T13:29:43.663 INFO:teuthology.orchestra.run.smithi076.stdout:5055 writesame 0x3c2088c thru 0x3c2950f data_size 0x2e(0x8c84 bytes) 2024-09-16T13:29:43.665 INFO:teuthology.orchestra.run.smithi076.stdout:5056 punch from 0x24f24e4 to 0x24f3a07, (0x1523 bytes) 2024-09-16T13:29:43.665 INFO:teuthology.orchestra.run.smithi076.stdout:5057 read 0x440b1c0 thru 0x4414296 (0x90d7 bytes) 2024-09-16T13:29:43.674 INFO:teuthology.orchestra.run.smithi076.stdout:5058 trunc from 0xc7579c2 to 0x4c5fb59 2024-09-16T13:29:43.681 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:43.680+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:43.696 INFO:teuthology.orchestra.run.smithi076.stdout:5061 writesame 0x7e31642 thru 0x7e3aeb0 data_size 0x1(0x986f bytes) 2024-09-16T13:29:43.710 INFO:teuthology.orchestra.run.smithi076.stdout:5062 write 0xd90745f thru 0xd90e1fb (0x6d9d bytes) 2024-09-16T13:29:43.713 INFO:teuthology.orchestra.run.smithi076.stdout:5064 writesame 0xc13053c thru 0xc1375fb data_size 0x58(0x70c0 bytes) 2024-09-16T13:29:43.714 INFO:teuthology.orchestra.run.smithi076.stdout:5065 punch from 0x33f1d22 to 0x33f4406, (0x26e4 bytes) 2024-09-16T13:29:43.715 INFO:teuthology.orchestra.run.smithi076.stdout:5066 writesame 0x5a96bfb thru 0x5a9980c data_size 0x1(0x2c12 bytes) 2024-09-16T13:29:43.723 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:43.723 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:43.723 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.724 INFO:teuthology.orchestra.run.smithi076.stdout:5068 punch from 0x204691e to 0x2049eb8, (0x359a bytes) 2024-09-16T13:29:43.724 INFO:teuthology.orchestra.run.smithi076.stdout:5069 punch from 0x8c6a8dc to 0x8c745ff, (0x9d23 bytes) 2024-09-16T13:29:43.724 INFO:teuthology.orchestra.run.smithi076.stdout:5070 punch from 0x248b603 to 0x2495f86, (0xa983 bytes) 2024-09-16T13:29:43.724 INFO:teuthology.orchestra.run.smithi076.stdout:5071 write 0xce898b2 thru 0xce9653d (0xcc8c bytes) 2024-09-16T13:29:43.729 INFO:teuthology.orchestra.run.smithi076.stdout:5073 write 0xd63275 thru 0xd70bc6 (0xd952 bytes) 2024-09-16T13:29:43.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5075 read 0x551f3b4 thru 0x55222d2 (0x2f1f bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5078 punch from 0xaac1c67 to 0xaac6609, (0x49a2 bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5079 punch from 0x76b0353 to 0x76b4980, (0x462d bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5081 read 0x9a84c69 thru 0x9a85bba (0xf52 bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5082 read 0xcc3f0db thru 0xcc44b42 (0x5a68 bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5083 punch from 0xd8e69be to 0xd8edc05, (0x7247 bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5084 read 0x8a5d6a7 thru 0x8a6c06e (0xe9c8 bytes) 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.734 INFO:teuthology.orchestra.run.smithi076.stdout:5087 punch from 0x517adac to 0x518279b, (0x79ef bytes) 2024-09-16T13:29:43.739 INFO:teuthology.orchestra.run.smithi076.stdout:5088 writesame 0xdfd88b2 thru 0xdfde86d data_size 0x16(0x5fbc bytes) 2024-09-16T13:29:43.740 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.740 INFO:teuthology.orchestra.run.smithi076.stdout:5090 read 0x2b06594 thru 0x2b15f53 (0xf9c0 bytes) 2024-09-16T13:29:43.745 INFO:teuthology.orchestra.run.smithi076.stdout:5091 trunc from 0xdfde86e to 0xa31f246 2024-09-16T13:29:43.754 INFO:teuthology.orchestra.run.smithi076.stdout:5092 write 0x5932e6 thru 0x596282 (0x2f9d bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5093 read 0x7bfdddc thru 0x7c0bcfd (0xdf22 bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5094 read 0x206cb4d thru 0x206fdba (0x326e bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5095 write 0xa504560 thru 0xa509575 (0x5016 bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5097 read 0x2d0d310 thru 0x2d0e2b6 (0xfa7 bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5099 trunc from 0xa509576 to 0x21da3dd 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5100 read 0x1a8094f thru 0x1a844b0 (0x3b62 bytes) 2024-09-16T13:29:43.818 INFO:teuthology.orchestra.run.smithi076.stdout:5101 trunc from 0x21da3dd to 0x9f5cd87 2024-09-16T13:29:43.819 INFO:teuthology.orchestra.run.smithi076.stdout:5102 read 0x681672b thru 0x6823531 (0xce07 bytes) 2024-09-16T13:29:43.819 INFO:teuthology.orchestra.run.smithi076.stdout:5106 read 0x52faf1a thru 0x52fd6bd (0x27a4 bytes) 2024-09-16T13:29:43.819 INFO:teuthology.orchestra.run.smithi076.stdout:5108 trunc from 0x9f5cd87 to 0xd637def 2024-09-16T13:29:43.819 INFO:teuthology.orchestra.run.smithi076.stdout:5110 write 0xb363b5f thru 0xb36e5f2 (0xaa94 bytes) 2024-09-16T13:29:43.819 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:43.815+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:43.829 INFO:teuthology.orchestra.run.smithi076.stdout:5112 trunc from 0xd637def to 0x20836d2 2024-09-16T13:29:43.860 INFO:teuthology.orchestra.run.smithi076.stdout:5113 write 0xec97f9e thru 0xeca437c (0xc3df bytes) 2024-09-16T13:29:43.863 INFO:teuthology.orchestra.run.smithi076.stdout:5114 read 0x1f34567 thru 0x1f424f9 (0xdf93 bytes) 2024-09-16T13:29:43.869 INFO:teuthology.orchestra.run.smithi076.stdout:5116 write 0xee5a87e thru 0xee5b1ed (0x970 bytes) 2024-09-16T13:29:43.872 INFO:teuthology.orchestra.run.smithi076.stdout:5117 punch from 0x81ee286 to 0x81f8996, (0xa710 bytes) 2024-09-16T13:29:43.872 INFO:teuthology.orchestra.run.smithi076.stdout:5118 read 0x28a1f8a thru 0x28b1dcf (0xfe46 bytes) 2024-09-16T13:29:43.873 INFO:teuthology.orchestra.run.smithi076.stdout:5119 punch from 0xcf9372d to 0xcf9a22c, (0x6aff bytes) 2024-09-16T13:29:43.873 INFO:teuthology.orchestra.run.smithi076.stdout:5120 punch from 0x2055d04 to 0x205e542, (0x883e bytes) 2024-09-16T13:29:43.873 INFO:teuthology.orchestra.run.smithi076.stdout:5121 punch from 0xb6cbdf to 0xb6ed51, (0x2172 bytes) 2024-09-16T13:29:43.873 INFO:teuthology.orchestra.run.smithi076.stdout:5122 writesame 0x8afbbfb thru 0x8b07af5 data_size 0x2b(0xbefb bytes) 2024-09-16T13:29:43.876 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.876 INFO:teuthology.orchestra.run.smithi076.stdout:5125 punch from 0xca5b17d to 0xca61fcc, (0x6e4f bytes) 2024-09-16T13:29:43.876 INFO:teuthology.orchestra.run.smithi076.stdout:5126 punch from 0x5eb1741 to 0x5ebdb5e, (0xc41d bytes) 2024-09-16T13:29:43.876 INFO:teuthology.orchestra.run.smithi076.stdout:5127 writesame 0x9f02c98 thru 0x9f060bb data_size 0x4(0x3424 bytes) 2024-09-16T13:29:43.880 INFO:teuthology.orchestra.run.smithi076.stdout:5129 read 0x4cbcca3 thru 0x4cc938a (0xc6e8 bytes) 2024-09-16T13:29:43.881 INFO:teuthology.orchestra.run.smithi076.stdout:5131 trunc from 0xee5b1ee to 0xe93daa9 2024-09-16T13:29:43.889 INFO:teuthology.orchestra.run.smithi076.stdout:5132 write 0xddef27a thru 0xddfaf06 (0xbc8d bytes) 2024-09-16T13:29:43.892 INFO:teuthology.orchestra.run.smithi076.stdout:5133 read 0x3f1b126 thru 0x3f2832d (0xd208 bytes) 2024-09-16T13:29:43.893 INFO:teuthology.orchestra.run.smithi076.stdout:5134 write 0x2b51dd2 thru 0x2b5b856 (0x9a85 bytes) 2024-09-16T13:29:43.896 INFO:teuthology.orchestra.run.smithi076.stdout:5135 read 0x84da0cf thru 0x84dd08a (0x2fbc bytes) 2024-09-16T13:29:43.896 INFO:teuthology.orchestra.run.smithi076.stdout:5136 write 0x834d066 thru 0x8359895 (0xc830 bytes) 2024-09-16T13:29:43.900 INFO:teuthology.orchestra.run.smithi076.stdout:5138 read 0x7afde1e thru 0x7b06352 (0x8535 bytes) 2024-09-16T13:29:43.900 INFO:teuthology.orchestra.run.smithi076.stdout:5139 read 0x531c453 thru 0x532bbe0 (0xf78e bytes) 2024-09-16T13:29:43.900 INFO:teuthology.orchestra.run.smithi076.stdout:5141 read 0x7177614 thru 0x717f042 (0x7a2f bytes) 2024-09-16T13:29:43.917 INFO:teuthology.orchestra.run.smithi076.stdout:5144 trunc from 0xe93daa9 to 0x49713b 2024-09-16T13:29:43.917 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:43.916+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:43.921 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:43.921 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:43.928 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.928 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.929 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.929 INFO:teuthology.orchestra.run.smithi076.stdout:5149 read 0xbcd72 thru 0xc864e (0xb8dd bytes) 2024-09-16T13:29:43.929 INFO:teuthology.orchestra.run.smithi076.stdout:5150 punch from 0x275ef8 to 0x27965e, (0x3766 bytes) 2024-09-16T13:29:43.930 INFO:teuthology.orchestra.run.smithi076.stdout:5151 trunc from 0x49713b to 0x81cd3c 2024-09-16T13:29:43.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:43.934 INFO:teuthology.orchestra.run.smithi076.stdout:5152 read 0x49f4f6 thru 0x4a55d1 (0x60dc bytes) 2024-09-16T13:29:43.945 INFO:teuthology.orchestra.run.smithi076.stdout:5153 trunc from 0x81cd3c to 0x8fac733 2024-09-16T13:29:43.950 INFO:teuthology.orchestra.run.smithi076.stdout:5154 write 0x6816411 thru 0x682416f (0xdd5f bytes) 2024-09-16T13:29:43.956 INFO:teuthology.orchestra.run.smithi076.stdout:5155 punch from 0x4210a50 to 0x421b803, (0xadb3 bytes) 2024-09-16T13:29:43.957 INFO:teuthology.orchestra.run.smithi076.stdout:5156 read 0x746acb1 thru 0x7475c1c (0xaf6c bytes) 2024-09-16T13:29:43.957 INFO:teuthology.orchestra.run.smithi076.stdout:5157 read 0x519f8af thru 0x51a4199 (0x48eb bytes) 2024-09-16T13:29:43.965 INFO:teuthology.orchestra.run.smithi076.stdout:5158 write 0xb9d944c thru 0xb9dab01 (0x16b6 bytes) 2024-09-16T13:29:43.968 INFO:teuthology.orchestra.run.smithi076.stdout:5159 read 0x74c9858 thru 0x74d1224 (0x79cd bytes) 2024-09-16T13:29:43.968 INFO:teuthology.orchestra.run.smithi076.stdout:5160 write 0x6bfccf4 thru 0x6c06d74 (0xa081 bytes) 2024-09-16T13:29:43.972 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:43.972 INFO:teuthology.orchestra.run.smithi076.stdout:5162 writesame 0x6d1e2a0 thru 0x6d23548 data_size 0x1(0x52a9 bytes) 2024-09-16T13:29:44.007 INFO:teuthology.orchestra.run.smithi076.stdout:5163 read 0x726429e thru 0x7266794 (0x24f7 bytes) 2024-09-16T13:29:44.007 INFO:teuthology.orchestra.run.smithi076.stdout:5166 punch from 0x9ff9345 to 0xa0023cd, (0x9088 bytes) 2024-09-16T13:29:44.007 INFO:teuthology.orchestra.run.smithi076.stdout:5167 punch from 0xad61479 to 0xad62c2b, (0x17b2 bytes) 2024-09-16T13:29:44.007 INFO:teuthology.orchestra.run.smithi076.stdout:5168 read 0x6de1389 thru 0x6de7490 (0x6108 bytes) 2024-09-16T13:29:44.007 INFO:teuthology.orchestra.run.smithi076.stdout:5169 punch from 0x746f1d0 to 0x747d866, (0xe696 bytes) 2024-09-16T13:29:44.008 INFO:teuthology.orchestra.run.smithi076.stdout:5173 punch from 0x77e5b2c to 0x77ee1ea, (0x86be bytes) 2024-09-16T13:29:44.008 INFO:teuthology.orchestra.run.smithi076.stdout:5174 write 0x5c43ba8 thru 0x5c48911 (0x4d6a bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5178 punch from 0x574084 to 0x579fbd, (0x5f39 bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5179 punch from 0x94fea29 to 0x9506efd, (0x84d4 bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5182 read 0xa65d409 thru 0xa66907f (0xbc77 bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5183 read 0x61a4aeb thru 0x61b2a15 (0xdf2b bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5184 punch from 0x80c41c3 to 0x80c65f9, (0x2436 bytes) 2024-09-16T13:29:44.010 INFO:teuthology.orchestra.run.smithi076.stdout:5185 read 0x27a9e2e thru 0x27aa284 (0x457 bytes) 2024-09-16T13:29:44.011 INFO:teuthology.orchestra.run.smithi076.stdout:5186 write 0xfc59a1 thru 0xfcda56 (0x80b6 bytes) 2024-09-16T13:29:44.012 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.012 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.014 INFO:teuthology.orchestra.run.smithi076.stdout:5189 trunc from 0xb9dab02 to 0xa9eeae5 2024-09-16T13:29:44.019 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:44.017+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:44.020 INFO:teuthology.orchestra.run.smithi076.stdout:5190 writesame 0x77991db thru 0x779ff21 data_size 0x19(0x6d47 bytes) 2024-09-16T13:29:44.022 INFO:teuthology.orchestra.run.smithi076.stdout:5191 read 0x1578963 thru 0x157a7c8 (0x1e66 bytes) 2024-09-16T13:29:44.022 INFO:teuthology.orchestra.run.smithi076.stdout:5192 punch from 0x6e1eb75 to 0x6e2a93a, (0xbdc5 bytes) 2024-09-16T13:29:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:5195 read 0x7b07cae thru 0x7b0cdbf (0x5112 bytes) 2024-09-16T13:29:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:5197 write 0xa7b4f89 thru 0xa7ba83d (0x58b5 bytes) 2024-09-16T13:29:44.027 INFO:teuthology.orchestra.run.smithi076.stdout:5198 punch from 0x13e2a39 to 0x13e33eb, (0x9b2 bytes) 2024-09-16T13:29:44.027 INFO:teuthology.orchestra.run.smithi076.stdout:5199 punch from 0x9603375 to 0x960b70a, (0x8395 bytes) 2024-09-16T13:29:44.027 INFO:teuthology.orchestra.run.smithi076.stdout:5200 punch from 0x2c35a18 to 0x2c3d5ab, (0x7b93 bytes) 2024-09-16T13:29:44.027 INFO:teuthology.orchestra.run.smithi076.stdout:5201 read 0x77de811 thru 0x77e4713 (0x5f03 bytes) 2024-09-16T13:29:44.027 INFO:teuthology.orchestra.run.smithi076.stdout:5206 write 0x149d95c thru 0x14a6ef9 (0x959e bytes) 2024-09-16T13:29:44.029 INFO:teuthology.orchestra.run.smithi076.stdout:5207 write 0x4a24704 thru 0x4a34279 (0xfb76 bytes) 2024-09-16T13:29:44.033 INFO:teuthology.orchestra.run.smithi076.stdout:5208 read 0x751ae5e thru 0x7525370 (0xa513 bytes) 2024-09-16T13:29:44.034 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.034 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.036 INFO:teuthology.orchestra.run.smithi076.stdout:5209 trunc from 0xa9eeae5 to 0x8929667 2024-09-16T13:29:44.042 INFO:teuthology.orchestra.run.smithi076.stdout:5211 writesame 0x100964e thru 0x100e4ac data_size 0x1(0x4e5f bytes) 2024-09-16T13:29:44.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:44.059 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.059 INFO:teuthology.orchestra.run.smithi076.stdout:5215 read 0xd4a7a2 thru 0xd4fade (0x533d bytes) 2024-09-16T13:29:44.060 INFO:teuthology.orchestra.run.smithi076.stdout:5217 read 0x632cc7c thru 0x63357f1 (0x8b76 bytes) 2024-09-16T13:29:44.069 INFO:teuthology.orchestra.run.smithi076.stdout:5218 write 0xafe34bf thru 0xafe58bf (0x2401 bytes) 2024-09-16T13:29:44.071 INFO:teuthology.orchestra.run.smithi076.stdout:5219 punch from 0x59d4a15 to 0x59e3a94, (0xf07f bytes) 2024-09-16T13:29:44.071 INFO:teuthology.orchestra.run.smithi076.stdout:5220 punch from 0x7ef7269 to 0x7f0698f, (0xf726 bytes) 2024-09-16T13:29:44.071 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.081 INFO:teuthology.orchestra.run.smithi076.stdout:5223 trunc from 0xafe58c0 to 0x29af03d 2024-09-16T13:29:44.111 INFO:teuthology.orchestra.run.smithi076.stdout:5224 write 0xd35db2e thru 0xd368345 (0xa818 bytes) 2024-09-16T13:29:44.114 INFO:teuthology.orchestra.run.smithi076.stdout:5225 read 0xb2c9c4e thru 0xb2cf922 (0x5cd5 bytes) 2024-09-16T13:29:44.115 INFO:teuthology.orchestra.run.smithi076.stdout:5226 writesame 0xaaf44ad thru 0xaaf5a6f data_size 0x3(0x15c3 bytes) 2024-09-16T13:29:44.118 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.119 INFO:teuthology.orchestra.run.smithi076.stdout:5228 trunc from 0xd368346 to 0xcf00029 2024-09-16T13:29:44.119 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:44.117+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:44.125 INFO:teuthology.orchestra.run.smithi076.stdout:5229 writesame 0x748ca23 thru 0x7492b00 data_size 0x3(0x60de bytes) 2024-09-16T13:29:44.134 INFO:teuthology.orchestra.run.smithi076.stdout:5230 write 0x46d2841 thru 0x46d6aaa (0x426a bytes) 2024-09-16T13:29:44.147 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.147 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.151 INFO:teuthology.orchestra.run.smithi076.stdout:5231 trunc from 0xcf00029 to 0x197f267 2024-09-16T13:29:44.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:44.164 INFO:teuthology.orchestra.run.smithi076.stdout:5232 read 0xc9b112 thru 0xcaaa08 (0xf8f7 bytes) 2024-09-16T13:29:44.164 INFO:teuthology.orchestra.run.smithi076.stdout:5233 punch from 0x1773c19 to 0x1775371, (0x1758 bytes) 2024-09-16T13:29:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:5234 trunc from 0x197f267 to 0x96bb01a 2024-09-16T13:29:44.180 INFO:teuthology.orchestra.run.smithi076.stdout:5235 punch from 0x7f6b430 to 0x7f6c9db, (0x15ab bytes) 2024-09-16T13:29:44.180 INFO:teuthology.orchestra.run.smithi076.stdout:5236 read 0x16b04bc thru 0x16b6fb5 (0x6afa bytes) 2024-09-16T13:29:44.180 INFO:teuthology.orchestra.run.smithi076.stdout:5240 write 0x8817dd0 thru 0x8818ff1 (0x1222 bytes) 2024-09-16T13:29:44.183 INFO:teuthology.orchestra.run.smithi076.stdout:5242 read 0x52aa94f thru 0x52b83db (0xda8d bytes) 2024-09-16T13:29:44.183 INFO:teuthology.orchestra.run.smithi076.stdout:5243 writesame 0x69ed411 thru 0x69f07dd data_size 0x1(0x33cd bytes) 2024-09-16T13:29:44.200 INFO:teuthology.orchestra.run.smithi076.stdout:5246 trunc from 0x96bb01a to 0x50b9a11 2024-09-16T13:29:44.212 INFO:teuthology.orchestra.run.smithi076.stdout:5248 punch from 0x4514a93 to 0x4524690, (0xfbfd bytes) 2024-09-16T13:29:44.219 INFO:teuthology.orchestra.run.smithi076.stdout:5249 write 0x6458c37 thru 0x6467172 (0xe53c bytes) 2024-09-16T13:29:44.234 INFO:teuthology.orchestra.run.smithi076.stdout:5250 write 0xd09ec45 thru 0xd0aa4f1 (0xb8ad bytes) 2024-09-16T13:29:44.237 INFO:teuthology.orchestra.run.smithi076.stdout:5252 read 0x89ff73a thru 0x8a0693a (0x7201 bytes) 2024-09-16T13:29:44.238 INFO:teuthology.orchestra.run.smithi076.stdout:5253 punch from 0x3ff9c20 to 0x4000bd4, (0x6fb4 bytes) 2024-09-16T13:29:44.241 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:44.239+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:44.242 INFO:teuthology.orchestra.run.smithi076.stdout:5257 write 0xd41dff7 thru 0xd424428 (0x6432 bytes) 2024-09-16T13:29:44.244 INFO:teuthology.orchestra.run.smithi076.stdout:5258 read 0xb394aab thru 0xb398dc3 (0x4319 bytes) 2024-09-16T13:29:44.244 INFO:teuthology.orchestra.run.smithi076.stdout:5259 read 0x5f86435 thru 0x5f86b66 (0x732 bytes) 2024-09-16T13:29:44.244 INFO:teuthology.orchestra.run.smithi076.stdout:5260 read 0x64f6ab8 thru 0x64fb738 (0x4c81 bytes) 2024-09-16T13:29:44.245 INFO:teuthology.orchestra.run.smithi076.stdout:5261 write 0xc4caf8d thru 0xc4d5001 (0xa075 bytes) 2024-09-16T13:29:44.247 INFO:teuthology.orchestra.run.smithi076.stdout:5262 punch from 0x1317d95 to 0x131efea, (0x7255 bytes) 2024-09-16T13:29:44.247 INFO:teuthology.orchestra.run.smithi076.stdout:5264 read 0x5465024 thru 0x5473204 (0xe1e1 bytes) 2024-09-16T13:29:44.247 INFO:teuthology.orchestra.run.smithi076.stdout:5265 read 0x70fe7eb thru 0x710ac62 (0xc478 bytes) 2024-09-16T13:29:44.262 INFO:teuthology.orchestra.run.smithi076.stdout:5267 trunc from 0xd424429 to 0x1a0fb0 2024-09-16T13:29:44.278 INFO:teuthology.orchestra.run.smithi076.stdout:5268 clone 41 order 23 su 65536 sc 8 2024-09-16T13:29:44.301 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.301 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:44.321 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:44.319+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:44.417 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:29:44.422 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:44.420+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:44.453 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.453 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:44.492 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone40 from 0x1a0fb0 (overlap 0x189575) to 0x9d562 2024-09-16T13:29:44.548 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:44.546+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:44.549 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-09-16T13:29:44.550 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.564 INFO:teuthology.orchestra.run.smithi076.stdout:5273 writesame 0x36f517b thru 0x36feeb7 data_size 0x1(0x9d3d bytes) 2024-09-16T13:29:44.570 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.570 INFO:teuthology.orchestra.run.smithi076.stdout:5275 writesame 0xc00d1a thru 0xc02845 data_size 0x2f(0x1b2c bytes) 2024-09-16T13:29:44.578 INFO:teuthology.orchestra.run.smithi076.stdout:5277 trunc from 0x36feeb8 to 0x307dc84 2024-09-16T13:29:44.603 INFO:teuthology.orchestra.run.smithi076.stdout:5278 trunc from 0x307dc84 to 0x9f88c37 2024-09-16T13:29:44.610 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.610 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.616 INFO:teuthology.orchestra.run.smithi076.stdout:5280 trunc from 0x9f88c37 to 0x4364055 2024-09-16T13:29:44.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:44.622 INFO:teuthology.orchestra.run.smithi076.stdout:5282 read 0x2c287e5 thru 0x2c36cc5 (0xe4e1 bytes) 2024-09-16T13:29:44.629 INFO:teuthology.orchestra.run.smithi076.stdout:5285 writesame 0x4b50672 thru 0x4b527a9 data_size 0x4(0x2138 bytes) 2024-09-16T13:29:44.629 INFO:teuthology.orchestra.run.smithi076.stdout:5286 read 0x4689b11 thru 0x468ad92 (0x1282 bytes) 2024-09-16T13:29:44.629 INFO:teuthology.orchestra.run.smithi076.stdout:5287 read 0x715684 thru 0x724383 (0xed00 bytes) 2024-09-16T13:29:44.645 INFO:teuthology.orchestra.run.smithi076.stdout:5290 write 0xc510cc3 thru 0xc51de01 (0xd13f bytes) 2024-09-16T13:29:44.648 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.648 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.648 INFO:teuthology.orchestra.run.smithi076.stdout:5296 punch from 0x6513c4f to 0x651eca7, (0xb058 bytes) 2024-09-16T13:29:44.648 INFO:teuthology.orchestra.run.smithi076.stdout:5297 writesame 0x20cd3c8 thru 0x20cf4ed data_size 0x1(0x2126 bytes) 2024-09-16T13:29:44.653 INFO:teuthology.orchestra.run.smithi076.stdout:5301 writesame 0x66fafc0 thru 0x6705d2c data_size 0x3(0xad6d bytes) 2024-09-16T13:29:44.660 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:44.658+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:44.672 INFO:teuthology.orchestra.run.smithi076.stdout:5302 read 0x551843e thru 0x5525620 (0xd1e3 bytes) 2024-09-16T13:29:44.672 INFO:teuthology.orchestra.run.smithi076.stdout:5303 punch from 0x1884213 to 0x189175d, (0xd54a bytes) 2024-09-16T13:29:44.672 INFO:teuthology.orchestra.run.smithi076.stdout:5304 write 0x77a63d6 thru 0x77aa2c6 (0x3ef1 bytes) 2024-09-16T13:29:44.675 INFO:teuthology.orchestra.run.smithi076.stdout:5305 punch from 0xaa5808b to 0xaa5b83e, (0x37b3 bytes) 2024-09-16T13:29:44.675 INFO:teuthology.orchestra.run.smithi076.stdout:5306 write 0x8812e75 thru 0x881aaa0 (0x7c2c bytes) 2024-09-16T13:29:44.680 INFO:teuthology.orchestra.run.smithi076.stdout:5307 read 0x1693d1f thru 0x16a1b09 (0xddeb bytes) 2024-09-16T13:29:44.681 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.681 INFO:teuthology.orchestra.run.smithi076.stdout:5310 write 0x9ea48c9 thru 0x9ea8265 (0x399d bytes) 2024-09-16T13:29:44.684 INFO:teuthology.orchestra.run.smithi076.stdout:5312 write 0x213ae9d thru 0x214691f (0xba83 bytes) 2024-09-16T13:29:44.686 INFO:teuthology.orchestra.run.smithi076.stdout:5313 writesame 0x691cf72 thru 0x6921cf4 data_size 0x1(0x4d83 bytes) 2024-09-16T13:29:44.713 INFO:teuthology.orchestra.run.smithi076.stdout:5314 writesame 0x78dfe85 thru 0x78e3373 data_size 0x1(0x34ef bytes) 2024-09-16T13:29:44.713 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.723 INFO:teuthology.orchestra.run.smithi076.stdout:5317 writesame 0xe873f62 thru 0xe879466 data_size 0x5(0x5505 bytes) 2024-09-16T13:29:44.723 INFO:teuthology.orchestra.run.smithi076.stdout:5318 write 0x746decb thru 0x7478589 (0xa6bf bytes) 2024-09-16T13:29:44.725 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:44.723+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:44.726 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.726 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.733 INFO:teuthology.orchestra.run.smithi076.stdout:5320 trunc from 0xe879467 to 0x928d6b9 2024-09-16T13:29:44.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:44.745 INFO:teuthology.orchestra.run.smithi076.stdout:5321 read 0x610ebd9 thru 0x61136c2 (0x4aea bytes) 2024-09-16T13:29:44.745 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.745 INFO:teuthology.orchestra.run.smithi076.stdout:5323 read 0x62601e1 thru 0x626c5f5 (0xc415 bytes) 2024-09-16T13:29:44.751 INFO:teuthology.orchestra.run.smithi076.stdout:5324 trunc from 0x928d6b9 to 0x4588431 2024-09-16T13:29:44.769 INFO:teuthology.orchestra.run.smithi076.stdout:5325 read 0x4070770 thru 0x4078fed (0x887e bytes) 2024-09-16T13:29:44.770 INFO:teuthology.orchestra.run.smithi076.stdout:5326 read 0xc8360 thru 0xd6151 (0xddf2 bytes) 2024-09-16T13:29:44.770 INFO:teuthology.orchestra.run.smithi076.stdout:5327 read 0x3be7d1 thru 0x3c8ba8 (0xa3d8 bytes) 2024-09-16T13:29:44.778 INFO:teuthology.orchestra.run.smithi076.stdout:5328 trunc from 0x4588431 to 0xa83cce4 2024-09-16T13:29:44.785 INFO:teuthology.orchestra.run.smithi076.stdout:5330 write 0x1a62b93 thru 0x1a72901 (0xfd6f bytes) 2024-09-16T13:29:44.789 INFO:teuthology.orchestra.run.smithi076.stdout:5331 read 0x7c7f3f2 thru 0x7c86a25 (0x7634 bytes) 2024-09-16T13:29:44.797 INFO:teuthology.orchestra.run.smithi076.stdout:5332 writesame 0xb9e8a24 thru 0xb9effb2 data_size 0xd(0x758f bytes) 2024-09-16T13:29:44.797 INFO:teuthology.orchestra.run.smithi076.stdout:5334 punch from 0xc8e5b2 to 0xc9a754, (0xc1a2 bytes) 2024-09-16T13:29:44.797 INFO:teuthology.orchestra.run.smithi076.stdout:5335 punch from 0x3ce9cb3 to 0x3cf33e1, (0x972e bytes) 2024-09-16T13:29:44.797 INFO:teuthology.orchestra.run.smithi076.stdout:5336 punch from 0xb75347b to 0xb76032d, (0xceb2 bytes) 2024-09-16T13:29:44.797 INFO:teuthology.orchestra.run.smithi076.stdout:5338 write 0x7fa62b9 thru 0x7fa7c97 (0x19df bytes) 2024-09-16T13:29:44.804 INFO:teuthology.orchestra.run.smithi076.stdout:5339 write 0xbd26e86 thru 0xbd335d8 (0xc753 bytes) 2024-09-16T13:29:44.808 INFO:teuthology.orchestra.run.smithi076.stdout:5340 read 0x1faf702 thru 0x1fb079f (0x109e bytes) 2024-09-16T13:29:44.808 INFO:teuthology.orchestra.run.smithi076.stdout:5341 read 0x56f3747 thru 0x56f7624 (0x3ede bytes) 2024-09-16T13:29:44.808 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.816 INFO:teuthology.orchestra.run.smithi076.stdout:5344 write 0xd2df83b thru 0xd2e1b8c (0x2352 bytes) 2024-09-16T13:29:44.821 INFO:teuthology.orchestra.run.smithi076.stdout:5348 punch from 0xc352145 to 0xc354b33, (0x29ee bytes) 2024-09-16T13:29:44.822 INFO:teuthology.orchestra.run.smithi076.stdout:5350 read 0x58d1484 thru 0x58d33b4 (0x1f31 bytes) 2024-09-16T13:29:44.822 INFO:teuthology.orchestra.run.smithi076.stdout:5352 write 0x6021790 thru 0x602d645 (0xbeb6 bytes) 2024-09-16T13:29:44.825 INFO:teuthology.orchestra.run.smithi076.stdout:5353 punch from 0xa774b24 to 0xa783043, (0xe51f bytes) 2024-09-16T13:29:44.825 INFO:teuthology.orchestra.run.smithi076.stdout:5354 punch from 0x63668f1 to 0x6369dd3, (0x34e2 bytes) 2024-09-16T13:29:44.825 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.825 INFO:teuthology.orchestra.run.smithi076.stdout:5356 read 0x2b5cfa4 thru 0x2b60782 (0x37df bytes) 2024-09-16T13:29:44.826 INFO:teuthology.orchestra.run.smithi076.stdout:5357 writesame 0xa9a8810 thru 0xa9b254a data_size 0x1(0x9d3b bytes) 2024-09-16T13:29:44.826 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:44.823+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:44.854 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.854 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:44.880 INFO:teuthology.orchestra.run.smithi076.stdout:5358 read 0xce30b78 thru 0xce33395 (0x281e bytes) 2024-09-16T13:29:44.882 INFO:teuthology.orchestra.run.smithi076.stdout:5359 trunc from 0xd2e1b8d to 0xe082c87 2024-09-16T13:29:44.888 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.888 INFO:teuthology.orchestra.run.smithi076.stdout:5361 read 0x1e7e79b thru 0x1e84674 (0x5eda bytes) 2024-09-16T13:29:44.888 INFO:teuthology.orchestra.run.smithi076.stdout:5363 punch from 0x41ddbd1 to 0x41eac29, (0xd058 bytes) 2024-09-16T13:29:44.888 INFO:teuthology.orchestra.run.smithi076.stdout:5364 read 0x2e18178 thru 0x2e20612 (0x849b bytes) 2024-09-16T13:29:44.889 INFO:teuthology.orchestra.run.smithi076.stdout:5366 writesame 0xa41a408 thru 0xa425d29 data_size 0x9(0xb922 bytes) 2024-09-16T13:29:44.894 INFO:teuthology.orchestra.run.smithi076.stdout:5367 write 0xdf7e032 thru 0xdf8bd79 (0xdd48 bytes) 2024-09-16T13:29:44.900 INFO:teuthology.orchestra.run.smithi076.stdout:5368 trunc from 0xe082c87 to 0xe25c429 2024-09-16T13:29:44.922 INFO:teuthology.orchestra.run.smithi076.stdout:5371 trunc from 0xe25c429 to 0x1be192 2024-09-16T13:29:44.925 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:44.924+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:44.953 INFO:teuthology.orchestra.run.smithi076.stdout:5373 writesame 0x6002def thru 0x600c93f data_size 0x1(0x9b51 bytes) 2024-09-16T13:29:44.954 INFO:teuthology.orchestra.run.smithi076.stdout:5374 read 0x5f1da1b thru 0x5f2690f (0x8ef5 bytes) 2024-09-16T13:29:44.956 INFO:teuthology.orchestra.run.smithi076.stdout:5375 trunc from 0x600c940 to 0x827bed0 2024-09-16T13:29:44.961 INFO:teuthology.orchestra.run.smithi076.stdout:5376 write 0x301eff4 thru 0x3025dd1 (0x6dde bytes) 2024-09-16T13:29:44.963 INFO:teuthology.orchestra.run.smithi076.stdout:5377 write 0x38ec675 thru 0x38f3061 (0x69ed bytes) 2024-09-16T13:29:44.966 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:44.966 INFO:teuthology.orchestra.run.smithi076.stdout:5381 punch from 0x968335 to 0x976ac5, (0xe790 bytes) 2024-09-16T13:29:44.966 INFO:teuthology.orchestra.run.smithi076.stdout:5382 read 0x6a31813 thru 0x6a37c1f (0x640d bytes) 2024-09-16T13:29:44.971 INFO:teuthology.orchestra.run.smithi076.stdout:5385 trunc from 0x827bed0 to 0x3b37d08 2024-09-16T13:29:44.994 INFO:teuthology.orchestra.run.smithi076.stdout:5386 write 0xb15c2fd thru 0xb163768 (0x746c bytes) 2024-09-16T13:29:44.995 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:44.995 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:44.997 INFO:teuthology.orchestra.run.smithi076.stdout:5387 write 0xb107b57 thru 0xb10cf60 (0x540a bytes) 2024-09-16T13:29:45.001 INFO:teuthology.orchestra.run.smithi076.stdout:5388 read 0x4cc346a thru 0x4ccddec (0xa983 bytes) 2024-09-16T13:29:45.001 INFO:teuthology.orchestra.run.smithi076.stdout:5389 write 0xa4585b7 thru 0xa4678e3 (0xf32d bytes) 2024-09-16T13:29:45.005 INFO:teuthology.orchestra.run.smithi076.stdout:5391 write 0x76ad1f1 thru 0x76b69bc (0x97cc bytes) 2024-09-16T13:29:45.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:45.014 INFO:teuthology.orchestra.run.smithi076.stdout:5392 punch from 0xa92caac to 0xa9374f5, (0xaa49 bytes) 2024-09-16T13:29:45.014 INFO:teuthology.orchestra.run.smithi076.stdout:5393 punch from 0x5955ea0 to 0x5964eb9, (0xf019 bytes) 2024-09-16T13:29:45.014 INFO:teuthology.orchestra.run.smithi076.stdout:5395 read 0x2f5eed2 thru 0x2f6d213 (0xe342 bytes) 2024-09-16T13:29:45.014 INFO:teuthology.orchestra.run.smithi076.stdout:5398 read 0x37b7dc5 thru 0x37c6ea4 (0xf0e0 bytes) 2024-09-16T13:29:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:5399 trunc from 0xb163769 to 0x2fd20f2 2024-09-16T13:29:45.026 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:45.025+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:45.040 INFO:teuthology.orchestra.run.smithi076.stdout:5400 read 0x143b8f6 thru 0x14431b1 (0x78bc bytes) 2024-09-16T13:29:45.061 INFO:teuthology.orchestra.run.smithi076.stdout:5401 write 0xdbba39d thru 0xdbc61be (0xbe22 bytes) 2024-09-16T13:29:45.065 INFO:teuthology.orchestra.run.smithi076.stdout:5402 clone 42 order 19 su 262144 sc 5 2024-09-16T13:29:45.118 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:45.118 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:45.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:45.127 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:45.126+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:45.229 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:45.227+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:45.265 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:45.265 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:45.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:45.329 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:45.327+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:45.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:45.372 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:45.373 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:45.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:45.466 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:45.464+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:45.470 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:45.470 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:45.487 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:45.495 INFO:teuthology.orchestra.run.smithi076.stdout:leaving image image_client.0-clone41 intact 2024-09-16T13:29:45.503 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:45.519 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:45.529 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:45.530 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:45.531 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:45.529+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:45.535 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:45.535 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:45.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:45.551 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:45.552 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:45.567 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:45.567 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:45.583 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:45.631 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:45.629+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:45.658 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:45.658 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:45.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:45.687 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-09-16T13:29:45.692 INFO:teuthology.orchestra.run.smithi076.stdout:5403 trunc from 0xdbc61bf to 0xde48597 2024-09-16T13:29:45.714 INFO:teuthology.orchestra.run.smithi076.stdout:5405 trunc from 0xde48597 to 0x26dd68b 2024-09-16T13:29:45.732 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:45.730+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:45.740 INFO:teuthology.orchestra.run.smithi076.stdout:5406 writesame 0x8c8a5dd thru 0x8c95ecd data_size 0x11(0xb8f1 bytes) 2024-09-16T13:29:45.740 INFO:teuthology.orchestra.run.smithi076.stdout:5407 write 0x440b7c thru 0x443cc4 (0x3149 bytes) 2024-09-16T13:29:45.743 INFO:teuthology.orchestra.run.smithi076.stdout:5409 read 0x53e56ea thru 0x53f1736 (0xc04d bytes) 2024-09-16T13:29:45.743 INFO:teuthology.orchestra.run.smithi076.stdout:5410 read 0x82e552c thru 0x82ef2e8 (0x9dbd bytes) 2024-09-16T13:29:45.749 INFO:teuthology.orchestra.run.smithi076.stdout:5411 write 0x90d3fe7 thru 0x90d5232 (0x124c bytes) 2024-09-16T13:29:45.753 INFO:teuthology.orchestra.run.smithi076.stdout:5412 read 0x8d310f7 thru 0x8d36a08 (0x5912 bytes) 2024-09-16T13:29:45.753 INFO:teuthology.orchestra.run.smithi076.stdout:5413 writesame 0x21988d8 thru 0x21a81b5 data_size 0x73(0xf8de bytes) 2024-09-16T13:29:45.757 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.757 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.757 INFO:teuthology.orchestra.run.smithi076.stdout:5418 punch from 0x2dc36a3 to 0x2dcb059, (0x79b6 bytes) 2024-09-16T13:29:45.758 INFO:teuthology.orchestra.run.smithi076.stdout:5420 write 0x656ec4d thru 0x6578afa (0x9eae bytes) 2024-09-16T13:29:45.769 INFO:teuthology.orchestra.run.smithi076.stdout:5421 trunc from 0x90d5233 to 0x27221f4 2024-09-16T13:29:45.788 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.653853319000007, 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.263035404999997, 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.625360936, 2024-09-16T13:29:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50252 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.545309731000003, 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092626319999999998, 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50252 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240262+0000", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2399999999999999e-07 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240264+0000", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5060000000000001e-06 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240282+0000", 2024-09-16T13:29:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8337000000000001e-05 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240286+0000", 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3900000000000002e-06 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249407+0000", 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091213830000000003 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249466+0000", 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8739999999999997e-05 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249525+0000", 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8952000000000001e-05 2024-09-16T13:29:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:50253 4.1c 4:38e45e03:::rbd_data.11a9441c6517.0000000000000000:15 [read 696320~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.545282411000002, 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093662180000000008, 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 50253 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240289+0000", 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240290+0000", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240292+0000", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240297+0000", 2024-09-16T13:29:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6790000000000002e-06 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.240301+0000", 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4230000000000001e-06 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249549+0000", 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092482740000000008 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249599+0000", 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0408000000000003e-05 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.249656+0000", 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6203999999999999e-05 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.470328899000002, 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.465033622, 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.464475063999998, 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53296 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 794624~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.791849775000003, 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093249630000000003, 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53296 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993722+0000", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5199999999999997e-07 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993725+0000", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6809999999999998e-06 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993734+0000", 2024-09-16T13:29:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4570000000000005e-06 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993738+0000", 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1230000000000002e-06 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002875+0000", 2024-09-16T13:29:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091369250000000006 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.002952+0000", 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7404999999999995e-05 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003047+0000", 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4820000000000004e-05 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.791821560999999, 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.791795315999998, 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.791775407999999, 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.215297405000001, 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.049949767000001, 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.033757022000003, 2024-09-16T13:29:45.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:45.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.907039121, 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:29:45.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:29:45.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.833208982, 2024-09-16T13:29:45.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:29:45.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.714467746, 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:29:45.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.702296057, 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:29:45.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:29:45.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.1475375579999998, 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:45.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:29:45.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:29:45.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:29:45.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:45.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:45.899 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:45.900 INFO:teuthology.orchestra.run.smithi076.stdout:5423 write 0x9718402 thru 0x9726498 (0xe097 bytes) 2024-09-16T13:29:45.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:5424 trunc from 0x9726499 to 0x8f2f34 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:5428 write 0xe3b99c0 thru 0xe3c39c2 (0xa003 bytes) 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:5429 trunc from 0xe3c39c3 to 0xc103235 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:5431 read 0xa1f9e93 thru 0xa1fcb7b (0x2ce9 bytes) 2024-09-16T13:29:45.902 INFO:teuthology.orchestra.run.smithi076.stdout:5434 writesame 0x1df37ea thru 0x1e015b8 data_size 0x1(0xddcf bytes) 2024-09-16T13:29:45.903 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:45.899+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:45.970 INFO:teuthology.orchestra.run.smithi076.stdout:5436 punch from 0x42b4185 to 0x42bf86f, (0xb6ea bytes) 2024-09-16T13:29:45.970 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.971 INFO:teuthology.orchestra.run.smithi076.stdout:5440 read 0x1730144 thru 0x173d395 (0xd252 bytes) 2024-09-16T13:29:45.971 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.971 INFO:teuthology.orchestra.run.smithi076.stdout:5444 read 0x55b9582 thru 0x55c66dc (0xd15b bytes) 2024-09-16T13:29:45.971 INFO:teuthology.orchestra.run.smithi076.stdout:5445 read 0x680aa16 thru 0x6818b98 (0xe183 bytes) 2024-09-16T13:29:45.971 INFO:teuthology.orchestra.run.smithi076.stdout:5446 write 0x6034b12 thru 0x60416d1 (0xcbc0 bytes) 2024-09-16T13:29:45.977 INFO:teuthology.orchestra.run.smithi076.stdout:5447 punch from 0x138d9f to 0x13d086, (0x42e7 bytes) 2024-09-16T13:29:45.978 INFO:teuthology.orchestra.run.smithi076.stdout:5448 writesame 0x6d2fbe5 thru 0x6d3fb9e data_size 0x9(0xffba bytes) 2024-09-16T13:29:45.986 INFO:teuthology.orchestra.run.smithi076.stdout:5450 punch from 0x6ea442e to 0x6eb2bf7, (0xe7c9 bytes) 2024-09-16T13:29:45.987 INFO:teuthology.orchestra.run.smithi076.stdout:5451 trunc from 0xc103235 to 0xccd1206 2024-09-16T13:29:45.993 INFO:teuthology.orchestra.run.smithi076.stdout:5452 read 0x51d32ea thru 0x51d8f54 (0x5c6b bytes) 2024-09-16T13:29:45.993 INFO:teuthology.orchestra.run.smithi076.stdout:5453 read 0x259432c thru 0x25a388f (0xf564 bytes) 2024-09-16T13:29:45.993 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.993 INFO:teuthology.orchestra.run.smithi076.stdout:5458 write 0x13d1d28 thru 0x13d1f36 (0x20f bytes) 2024-09-16T13:29:45.996 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:45.996 INFO:teuthology.orchestra.run.smithi076.stdout:5461 write 0xb7654b6 thru 0xb767390 (0x1edb bytes) 2024-09-16T13:29:45.998 INFO:teuthology.orchestra.run.smithi076.stdout:5463 write 0x486ce2 thru 0x4933ed (0xc70c bytes) 2024-09-16T13:29:46.000 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:45.999+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:46.009 INFO:teuthology.orchestra.run.smithi076.stdout:5464 write 0xd14365b thru 0xd14f519 (0xbebf bytes) 2024-09-16T13:29:46.021 INFO:teuthology.orchestra.run.smithi076.stdout:5466 trunc from 0xd14f51a to 0x5abdea2 2024-09-16T13:29:46.032 INFO:teuthology.orchestra.run.smithi076.stdout:5468 read 0x172d08d thru 0x172f0f7 (0x206b bytes) 2024-09-16T13:29:46.032 INFO:teuthology.orchestra.run.smithi076.stdout:5469 punch from 0x3830c1b to 0x383af86, (0xa36b bytes) 2024-09-16T13:29:46.041 INFO:teuthology.orchestra.run.smithi076.stdout:5470 trunc from 0x5abdea2 to 0xd6ace12 2024-09-16T13:29:46.051 INFO:teuthology.orchestra.run.smithi076.stdout:5472 trunc from 0xd6ace12 to 0x932c5d6 2024-09-16T13:29:46.052 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:46.052 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:46.052 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:46.052 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:46.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:46.067 INFO:teuthology.orchestra.run.smithi076.stdout:5474 write 0xd74255a thru 0xd74cbba (0xa661 bytes) 2024-09-16T13:29:46.071 INFO:teuthology.orchestra.run.smithi076.stdout:5475 punch from 0x17eb12f to 0x17f84f1, (0xd3c2 bytes) 2024-09-16T13:29:46.071 INFO:teuthology.orchestra.run.smithi076.stdout:5476 write 0x6a0d6a8 thru 0x6a1530a (0x7c63 bytes) 2024-09-16T13:29:46.076 INFO:teuthology.orchestra.run.smithi076.stdout:5477 write 0x25aa46f thru 0x25b4eae (0xaa40 bytes) 2024-09-16T13:29:46.081 INFO:teuthology.orchestra.run.smithi076.stdout:5480 punch from 0x7bca1b to 0x7c7d14, (0xb2f9 bytes) 2024-09-16T13:29:46.081 INFO:teuthology.orchestra.run.smithi076.stdout:5481 punch from 0xaa4cb4 to 0xab18b1, (0xcbfd bytes) 2024-09-16T13:29:46.081 INFO:teuthology.orchestra.run.smithi076.stdout:5482 write 0xbd1dea9 thru 0xbd2211e (0x4276 bytes) 2024-09-16T13:29:46.085 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.085 INFO:teuthology.orchestra.run.smithi076.stdout:5484 read 0x852e7ce thru 0x8535fa8 (0x77db bytes) 2024-09-16T13:29:46.085 INFO:teuthology.orchestra.run.smithi076.stdout:5485 punch from 0x80a2eb to 0x810335, (0x604a bytes) 2024-09-16T13:29:46.085 INFO:teuthology.orchestra.run.smithi076.stdout:5486 flatten 2024-09-16T13:29:46.101 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:46.099+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:46.106 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.106 INFO:teuthology.orchestra.run.smithi076.stdout:5488 punch from 0xcfab6b0 to 0xcfb5198, (0x9ae8 bytes) 2024-09-16T13:29:46.106 INFO:teuthology.orchestra.run.smithi076.stdout:5489 write 0x5d25560 thru 0x5d2cd21 (0x77c2 bytes) 2024-09-16T13:29:46.111 INFO:teuthology.orchestra.run.smithi076.stdout:5490 punch from 0x4f83285 to 0x4f8fec6, (0xcc41 bytes) 2024-09-16T13:29:46.111 INFO:teuthology.orchestra.run.smithi076.stdout:5491 write 0x9bc6c81 thru 0x9bd1ebf (0xb23f bytes) 2024-09-16T13:29:46.116 INFO:teuthology.orchestra.run.smithi076.stdout:5492 write 0xe90d8e thru 0xe93ffc (0x326f bytes) 2024-09-16T13:29:46.120 INFO:teuthology.orchestra.run.smithi076.stdout:5493 read 0x38b051d thru 0x38b7648 (0x712c bytes) 2024-09-16T13:29:46.120 INFO:teuthology.orchestra.run.smithi076.stdout:5494 write 0xd69f8e0 thru 0xd6a0c1c (0x133d bytes) 2024-09-16T13:29:46.123 INFO:teuthology.orchestra.run.smithi076.stdout:5495 write 0x3e3a937 thru 0x3e3d546 (0x2c10 bytes) 2024-09-16T13:29:46.126 INFO:teuthology.orchestra.run.smithi076.stdout:5496 write 0x5f97433 thru 0x5f9759b (0x169 bytes) 2024-09-16T13:29:46.129 INFO:teuthology.orchestra.run.smithi076.stdout:5497 read 0x98fdb47 thru 0x9901b69 (0x4023 bytes) 2024-09-16T13:29:46.129 INFO:teuthology.orchestra.run.smithi076.stdout:5498 read 0xc33dcc4 thru 0xc33e89a (0xbd7 bytes) 2024-09-16T13:29:46.129 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.129 INFO:teuthology.orchestra.run.smithi076.stdout:5502 read 0x85a92a4 thru 0x85b726c (0xdfc9 bytes) 2024-09-16T13:29:46.145 INFO:teuthology.orchestra.run.smithi076.stdout:5503 trunc from 0xd74cbbb to 0x239429 2024-09-16T13:29:46.162 INFO:teuthology.orchestra.run.smithi076.stdout:5504 read 0x160ffe thru 0x1673c0 (0x63c3 bytes) 2024-09-16T13:29:46.162 INFO:teuthology.orchestra.run.smithi076.stdout:5505 read 0x178d0f thru 0x185c01 (0xcef3 bytes) 2024-09-16T13:29:46.162 INFO:teuthology.orchestra.run.smithi076.stdout:5508 read 0xfd6fe thru 0xfebed (0x14f0 bytes) 2024-09-16T13:29:46.163 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.163 INFO:teuthology.orchestra.run.smithi076.stdout:5511 trunc from 0x239429 to 0x2a9376 2024-09-16T13:29:46.180 INFO:teuthology.orchestra.run.smithi076.stdout:5512 writesame 0x7013faa thru 0x7020dd1 data_size 0x24(0xce28 bytes) 2024-09-16T13:29:46.181 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:46.181 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:46.181 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:46.181 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:46.181 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:46.183 INFO:teuthology.orchestra.run.smithi076.stdout:5513 trunc from 0x7020dd2 to 0x9b7c41f 2024-09-16T13:29:46.190 INFO:teuthology.orchestra.run.smithi076.stdout:5515 writesame 0xf7f590 thru 0xf86ec4 data_size 0x1(0x7935 bytes) 2024-09-16T13:29:46.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:46.202 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:46.200+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:46.240 INFO:teuthology.orchestra.run.smithi076.stdout:5516 write 0x2bbd626 thru 0x2bbf941 (0x231c bytes) 2024-09-16T13:29:46.255 INFO:teuthology.orchestra.run.smithi076.stdout:5517 write 0xe6c3828 thru 0xe6c7467 (0x3c40 bytes) 2024-09-16T13:29:46.258 INFO:teuthology.orchestra.run.smithi076.stdout:5519 read 0x1c9928f thru 0x1ca4f0d (0xbc7f bytes) 2024-09-16T13:29:46.258 INFO:teuthology.orchestra.run.smithi076.stdout:5520 write 0x46c16da thru 0x46c97de (0x8105 bytes) 2024-09-16T13:29:46.262 INFO:teuthology.orchestra.run.smithi076.stdout:5521 writesame 0x4e17a88 thru 0x4e19643 data_size 0x32(0x1bbc bytes) 2024-09-16T13:29:46.265 INFO:teuthology.orchestra.run.smithi076.stdout:5522 write 0xc85d79e thru 0xc85ea77 (0x12da bytes) 2024-09-16T13:29:46.268 INFO:teuthology.orchestra.run.smithi076.stdout:5523 writesame 0x6cd9c66 thru 0x6ce0402 data_size 0x5(0x679d bytes) 2024-09-16T13:29:46.275 INFO:teuthology.orchestra.run.smithi076.stdout:5524 writesame 0xd77f2e8 thru 0xd78b879 data_size 0x26(0xc592 bytes) 2024-09-16T13:29:46.279 INFO:teuthology.orchestra.run.smithi076.stdout:5525 read 0x5367cbf thru 0x536a649 (0x298b bytes) 2024-09-16T13:29:46.279 INFO:teuthology.orchestra.run.smithi076.stdout:5526 write 0x445727d thru 0x4459d0b (0x2a8f bytes) 2024-09-16T13:29:46.283 INFO:teuthology.orchestra.run.smithi076.stdout:5527 writesame 0x4e8b8fc thru 0x4e8beaa data_size 0x5(0x5af bytes) 2024-09-16T13:29:46.289 INFO:teuthology.orchestra.run.smithi076.stdout:5529 read 0x552160f thru 0x552ce8f (0xb881 bytes) 2024-09-16T13:29:46.289 INFO:teuthology.orchestra.run.smithi076.stdout:5531 read 0x41de6b1 thru 0x41df848 (0x1198 bytes) 2024-09-16T13:29:46.289 INFO:teuthology.orchestra.run.smithi076.stdout:5532 writesame 0x750d12e thru 0x7518fac data_size 0x1(0xbe7f bytes) 2024-09-16T13:29:46.303 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:46.301+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091560878999999, 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:29:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091498405999999, 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:29:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:29:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091453753000003, 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:29:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:29:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091380651999998, 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:29:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:29:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091362146000002, 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:29:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:29:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091188193999997, 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:29:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.091149639999998, 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:29:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.090190946, 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:29:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.090133149000003, 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:29:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:29:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.090039273000002, 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:29:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.090011283999999, 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:29:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:29:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.088737258000002, 2024-09-16T13:29:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:29:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.088714011999997, 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:29:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:29:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.361367862999998, 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:29:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:29:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.056739961000002, 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:29:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:29:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:29:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.044019130999999, 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:29:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 21.820985907000001, 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 18.341412828999999, 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:29:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:29:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.46369645199999998, 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:29:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:29:46.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:29:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.44656215199999999, 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:46.457 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:46.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:29:46.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5533 writesame 0x1155579 thru 0x115b493 data_size 0x1(0x5f1b bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5535 punch from 0xd3f5336 to 0xd401401, (0xc0cb bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5536 read 0xc5d2cc5 thru 0xc5d556a (0x28a6 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5537 read 0x6fdff79 thru 0x6fed1d8 (0xd260 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5541 read 0xd318a66 thru 0xd3223de (0x9979 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5542 read 0xd6209cb thru 0xd62a5ca (0x9c00 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5544 write 0x2412e08 thru 0x241a40e (0x7607 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5545 read 0xd2e2947 thru 0xd2eb7e8 (0x8ea2 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5546 writesame 0x248cf78 thru 0x249052e data_size 0x1(0x35b7 bytes) 2024-09-16T13:29:46.460 INFO:teuthology.orchestra.run.smithi076.stdout:5547 trunc from 0xe6c7468 to 0x5341d52 2024-09-16T13:29:46.461 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:46.456+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:46.466 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.466 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.475 INFO:teuthology.orchestra.run.smithi076.stdout:5553 trunc from 0x5341d52 to 0xd6cd81b 2024-09-16T13:29:46.481 INFO:teuthology.orchestra.run.smithi076.stdout:5554 writesame 0xaebe012 thru 0xaec6abe data_size 0x1(0x8aad bytes) 2024-09-16T13:29:46.535 INFO:teuthology.orchestra.run.smithi076.stdout:5555 trunc from 0xd6cd81b to 0xd353934 2024-09-16T13:29:46.540 INFO:teuthology.orchestra.run.smithi076.stdout:5557 writesame 0x3c6ea3c thru 0x3c6ffba data_size 0x1(0x157f bytes) 2024-09-16T13:29:46.545 INFO:teuthology.orchestra.run.smithi076.stdout:5559 read 0x9eee69a thru 0x9efd1df (0xeb46 bytes) 2024-09-16T13:29:46.545 INFO:teuthology.orchestra.run.smithi076.stdout:5560 write 0x64ea087 thru 0x64f4fa3 (0xaf1d bytes) 2024-09-16T13:29:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.554 INFO:teuthology.orchestra.run.smithi076.stdout:5563 write 0x545b8f1 thru 0x546349e (0x7bae bytes) 2024-09-16T13:29:46.558 INFO:teuthology.orchestra.run.smithi076.stdout:5564 punch from 0x62e0930 to 0x62efcfb, (0xf3cb bytes) 2024-09-16T13:29:46.558 INFO:teuthology.orchestra.run.smithi076.stdout:5566 write 0x5495afa thru 0x54a5225 (0xf72c bytes) 2024-09-16T13:29:46.558 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:46.556+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:103911 4.5 e99/32 4:a2d62da1:::rbd_data.11a976e61a66.00000000000000a8:head v 99'490, mlcod=99'488)", 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.559207+0000", 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.001814902, 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001830789, 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.562 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559207+0000", 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559207+0000", 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.563 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559208+0000", 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0399999999999995e-07 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559233+0000", 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5106999999999999e-05 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559245+0000", 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1939e-05 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.564 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559250+0000", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.028e-06 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559291+0000", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1570999999999999e-05 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.559307+0000", 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6027000000000001e-05 2024-09-16T13:29:46.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:46.566 INFO:teuthology.orchestra.run.smithi076.stdout:5567 read 0x59437e3 thru 0x59489cb (0x51e9 bytes) 2024-09-16T13:29:46.567 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.567 INFO:teuthology.orchestra.run.smithi076.stdout:5569 trunc from 0xd353934 to 0xe6da976 2024-09-16T13:29:46.571 INFO:teuthology.orchestra.run.smithi076.stdout:5573 punch from 0xad5531c to 0xad5a818, (0x54fc bytes) 2024-09-16T13:29:46.571 INFO:teuthology.orchestra.run.smithi076.stdout:5574 read 0x33824ac thru 0x338c388 (0x9edd bytes) 2024-09-16T13:29:46.571 INFO:teuthology.orchestra.run.smithi076.stdout:5575 read 0x61d3258 thru 0x61d947d (0x6226 bytes) 2024-09-16T13:29:46.571 INFO:teuthology.orchestra.run.smithi076.stdout:5577 write 0xb1f10ad thru 0xb1faa32 (0x9986 bytes) 2024-09-16T13:29:46.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:46.575 INFO:teuthology.orchestra.run.smithi076.stdout:5578 writesame 0x1ffa8f4 thru 0x200a6c9 data_size 0x1(0xfdd6 bytes) 2024-09-16T13:29:46.650 INFO:teuthology.orchestra.run.smithi076.stdout:5580 write 0xdb0a6bb thru 0xdb0bc3c (0x1582 bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5581 read 0x8510d35 thru 0x8519bdf (0x8eab bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5583 read 0x2fc1b20 thru 0x2fc4149 (0x262a bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5584 read 0xbe760f3 thru 0xbe77121 (0x102f bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5585 read 0x6d1a16c thru 0x6d24cdf (0xab74 bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5587 read 0x3dc103c thru 0x3dc9f40 (0x8f05 bytes) 2024-09-16T13:29:46.654 INFO:teuthology.orchestra.run.smithi076.stdout:5588 punch from 0xce0b847 to 0xce16390, (0xab49 bytes) 2024-09-16T13:29:46.655 INFO:teuthology.orchestra.run.smithi076.stdout:5590 punch from 0xdb9f236 to 0xdba471c, (0x54e6 bytes) 2024-09-16T13:29:46.655 INFO:teuthology.orchestra.run.smithi076.stdout:5591 read 0xbd863c6 thru 0xbd959b1 (0xf5ec bytes) 2024-09-16T13:29:46.655 INFO:teuthology.orchestra.run.smithi076.stdout:5592 read 0x19d96bb thru 0x19e249b (0x8de1 bytes) 2024-09-16T13:29:46.655 INFO:teuthology.orchestra.run.smithi076.stdout:5593 writesame 0x3301047 thru 0x330b753 data_size 0x5(0xa70d bytes) 2024-09-16T13:29:46.660 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:46.657+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:46.667 INFO:teuthology.orchestra.run.smithi076.stdout:5594 writesame 0xe9ee80c thru 0xe9fd553 data_size 0xc(0xed48 bytes) 2024-09-16T13:29:46.667 INFO:teuthology.orchestra.run.smithi076.stdout:5596 punch from 0xd67a1d to 0xd6f54f, (0x7b32 bytes) 2024-09-16T13:29:46.680 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:46.680 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:46.681 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:46.684 INFO:teuthology.orchestra.run.smithi076.stdout:5597 trunc from 0xe9fd554 to 0xdb76f9 2024-09-16T13:29:46.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:46.710 INFO:teuthology.orchestra.run.smithi076.stdout:5598 write 0x2d32d25 thru 0x2d381f3 (0x54cf bytes) 2024-09-16T13:29:46.714 INFO:teuthology.orchestra.run.smithi076.stdout:5599 punch from 0xf7a055 to 0xf7b056, (0x1001 bytes) 2024-09-16T13:29:46.714 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.714 INFO:teuthology.orchestra.run.smithi076.stdout:5603 trunc from 0x2d381f4 to 0x3144354 2024-09-16T13:29:46.718 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:46.722 INFO:teuthology.orchestra.run.smithi076.stdout:5605 trunc from 0x3144354 to 0x5fa2796 2024-09-16T13:29:46.740 INFO:teuthology.orchestra.run.smithi076.stdout:5607 writesame 0xcbc1641 thru 0xcbc1d18 data_size 0xc(0x6d8 bytes) 2024-09-16T13:29:46.740 INFO:teuthology.orchestra.run.smithi076.stdout:5608 punch from 0x506bc97 to 0x5070520, (0x4889 bytes) 2024-09-16T13:29:46.740 INFO:teuthology.orchestra.run.smithi076.stdout:5610 write 0x4575125 thru 0x457ecde (0x9bba bytes) 2024-09-16T13:29:46.744 INFO:teuthology.orchestra.run.smithi076.stdout:5611 read 0x8425c45 thru 0x8427eeb (0x22a7 bytes) 2024-09-16T13:29:46.744 INFO:teuthology.orchestra.run.smithi076.stdout:5612 read 0xc38827b thru 0xc397a94 (0xf81a bytes) 2024-09-16T13:29:46.744 INFO:teuthology.orchestra.run.smithi076.stdout:5613 writesame 0x6e116a7 thru 0x6e19cc4 data_size 0x1(0x861e bytes) 2024-09-16T13:29:46.759 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:46.758+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:46.807 INFO:teuthology.orchestra.run.smithi076.stdout:5616 writesame 0xaeed716 thru 0xaef157f data_size 0x3(0x3e6a bytes) 2024-09-16T13:29:46.813 INFO:teuthology.orchestra.run.smithi076.stdout:5618 read 0x64fb645 thru 0x650126f (0x5c2b bytes) 2024-09-16T13:29:46.813 INFO:teuthology.orchestra.run.smithi076.stdout:5619 clone 43 order 18 su 32768 sc 2 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:46.830 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.116961313999994, 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:46.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:46.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.107422202999999, 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:46.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:46.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.140965467999997, 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:46.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:46.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.430554016999999, 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:46.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.419906967999999, 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:46.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.856 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.752845387000001, 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.857 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:46.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:46.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:46.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:46.862 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.185214331000001, 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:29:46.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:29:46.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.168081437000001, 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.871 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.207692491, 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.191203770000001, 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:29:46.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.811318733, 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.881 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:46.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:29:46.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.885 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.810725804, 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:29:46.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:29:46.889 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.810711141000001, 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:29:46.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:29:46.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.5161344890000006, 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.894 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:46.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.5028532630000004, 2024-09-16T13:29:46.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:29:46.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.53760358799999997, 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:29:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.52132685899999998, 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:29:46.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.914 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.24998194500000001, 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:29:46.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:29:46.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.920 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.24978705000000001, 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.921 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:29:46.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.926 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.23052413199999999, 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:46.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:29:46.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:29:46.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:46.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:46.931 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:46.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:46.933 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:46.931+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:47.032 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:47.032 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:47.032 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:47.032 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:47.033 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:47.031+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:47.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:47.134 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:47.133+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:47.178 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:47.178 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:47.178 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:47.178 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:47.178 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:47.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:47.271 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:47.269+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.594277132000002, 2024-09-16T13:29:47.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:47.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:47.333 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 70.151385809999994, 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:47.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:47.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:47.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.637927781999998, 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:47.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:47.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.627838515000001, 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:47.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.349125487999999, 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:47.348 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:47.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.400966515, 2024-09-16T13:29:47.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:29:47.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:47.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:29:47.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.31378653, 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.362 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:29:47.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:47.366 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313768932, 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:29:47.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313754232999999, 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.371 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:29:47.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:29:47.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313743302000001, 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:47.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.380 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313727683, 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:29:47.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313706420000001, 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.386 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:29:47.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.390 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313691410000001, 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:29:47.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:29:47.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:29:47.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.395 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313675329000001, 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:29:47.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:29:47.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313635462000001, 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:29:47.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.313624109999999, 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.405 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:29:47.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:47.409 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.5890732239999998, 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:47.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:29:47.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.5877848979999998, 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:47.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:29:47.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:29:47.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.587512963, 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:29:47.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.582034341, 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:47.425 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:29:47.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:47.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:47.430 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:47.430 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:29:47.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:47.432 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:47.429+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:47.531 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:47.530+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:47.571 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:47.572 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:47.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:47.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:47.635 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #41, image image_client.0-clone41 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent42 2024-09-16T13:29:47.635 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:47.633+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:47.719 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:47.719 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:47.719 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:47.719 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:47.720 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:47.724 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:47.724 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:47.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:47.733 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:47.732+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:47.740 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:47.740 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:47.756 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:47.756 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:47.772 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:47.772 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:47.787 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:47.803 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:47.803 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:47.819 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:47.819 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:47.826 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:47.826 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:47.835 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:47.833+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:47.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:47.836 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:47.836 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:47.882 INFO:teuthology.orchestra.run.smithi076.stdout:5620 read 0x90759f6 thru 0x907fcd1 (0xa2dc bytes) 2024-09-16T13:29:47.883 INFO:teuthology.orchestra.run.smithi076.stdout:5621 read 0x1d6484 thru 0x1d67e2 (0x35f bytes) 2024-09-16T13:29:47.887 INFO:teuthology.orchestra.run.smithi076.stdout:5622 trunc from 0xcbc1d19 to 0x9ff93be 2024-09-16T13:29:47.898 INFO:teuthology.orchestra.run.smithi076.stdout:5623 punch from 0x7e2e452 to 0x7e30a1f, (0x25cd bytes) 2024-09-16T13:29:47.898 INFO:teuthology.orchestra.run.smithi076.stdout:5624 read 0x5eed9c thru 0x5f532a (0x658f bytes) 2024-09-16T13:29:47.908 INFO:teuthology.orchestra.run.smithi076.stdout:5626 writesame 0xd38c538 thru 0xd38e048 data_size 0x1(0x1b11 bytes) 2024-09-16T13:29:47.908 INFO:teuthology.orchestra.run.smithi076.stdout:5629 read 0x7cb76b1 thru 0x7cc46f0 (0xd040 bytes) 2024-09-16T13:29:47.908 INFO:teuthology.orchestra.run.smithi076.stdout:5630 punch from 0x8180e73 to 0x818fc88, (0xee15 bytes) 2024-09-16T13:29:47.908 INFO:teuthology.orchestra.run.smithi076.stdout:5631 write 0x727c3b1 thru 0x727dc44 (0x1894 bytes) 2024-09-16T13:29:47.911 INFO:teuthology.orchestra.run.smithi076.stdout:5633 writesame 0x4b6387f thru 0x4b72200 data_size 0x7b(0xe982 bytes) 2024-09-16T13:29:47.917 INFO:teuthology.orchestra.run.smithi076.stdout:5634 writesame 0x26e067d thru 0x26e5e21 data_size 0x1b(0x57a5 bytes) 2024-09-16T13:29:47.927 INFO:teuthology.orchestra.run.smithi076.stdout:5636 trunc from 0xd38e049 to 0x7d5fee3 2024-09-16T13:29:47.936 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:47.934+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:47.940 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:47.940 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:47.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:47.953 INFO:teuthology.orchestra.run.smithi076.stdout:5639 trunc from 0x7d5fee3 to 0x20240a7 2024-09-16T13:29:47.979 INFO:teuthology.orchestra.run.smithi076.stdout:5642 write 0x6694dfa thru 0x6697b74 (0x2d7b bytes) 2024-09-16T13:29:47.982 INFO:teuthology.orchestra.run.smithi076.stdout:5643 write 0x23a5c53 thru 0x23a6f90 (0x133e bytes) 2024-09-16T13:29:47.995 INFO:teuthology.orchestra.run.smithi076.stdout:5644 write 0xa7bc6c0 thru 0xa7c9b6c (0xd4ad bytes) 2024-09-16T13:29:47.999 INFO:teuthology.orchestra.run.smithi076.stdout:5645 punch from 0x8542dc6 to 0x8545464, (0x269e bytes) 2024-09-16T13:29:47.999 INFO:teuthology.orchestra.run.smithi076.stdout:5646 read 0x18b253a thru 0x18b39b3 (0x147a bytes) 2024-09-16T13:29:47.999 INFO:teuthology.orchestra.run.smithi076.stdout:5647 writesame 0x3271f83 thru 0x327df30 data_size 0x23(0xbfae bytes) 2024-09-16T13:29:48.003 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.003 INFO:teuthology.orchestra.run.smithi076.stdout:5649 write 0x627f81b thru 0x6282487 (0x2c6d bytes) 2024-09-16T13:29:48.007 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.007 INFO:teuthology.orchestra.run.smithi076.stdout:5652 write 0x3942662 thru 0x394a6fd (0x809c bytes) 2024-09-16T13:29:48.011 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.016 INFO:teuthology.orchestra.run.smithi076.stdout:5656 trunc from 0xa7c9b6d to 0x5f7d116 2024-09-16T13:29:48.025 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.026 INFO:teuthology.orchestra.run.smithi076.stdout:5658 write 0x2516dc2 thru 0x251eb71 (0x7db0 bytes) 2024-09-16T13:29:48.037 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:48.035+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:48.043 INFO:teuthology.orchestra.run.smithi076.stdout:5659 writesame 0xc301e5e thru 0xc30c137 data_size 0x16(0xa2da bytes) 2024-09-16T13:29:48.045 INFO:teuthology.orchestra.run.smithi076.stdout:5661 trunc from 0xc30c138 to 0xd2ffc5d 2024-09-16T13:29:48.050 INFO:teuthology.orchestra.run.smithi076.stdout:5662 writesame 0x26e6d71 thru 0x26f5382 data_size 0x31(0xe612 bytes) 2024-09-16T13:29:48.061 INFO:teuthology.orchestra.run.smithi076.stdout:5664 trunc from 0xd2ffc5d to 0x9ae2f7b 2024-09-16T13:29:48.078 INFO:teuthology.orchestra.run.smithi076.stdout:5667 write 0xed2cfc6 thru 0xed3537a (0x83b5 bytes) 2024-09-16T13:29:48.091 INFO:teuthology.orchestra.run.smithi076.stdout:5668 trunc from 0xed3537b to 0x79c2206 2024-09-16T13:29:48.096 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.096 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.100 INFO:teuthology.orchestra.run.smithi076.stdout:5669 punch from 0x3510c54 to 0x3513438, (0x27e4 bytes) 2024-09-16T13:29:48.100 INFO:teuthology.orchestra.run.smithi076.stdout:5671 punch from 0x271a54d to 0x27285b7, (0xe06a bytes) 2024-09-16T13:29:48.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:48.114 INFO:teuthology.orchestra.run.smithi076.stdout:5674 writesame 0xedd0f18 thru 0xedda118 data_size 0x3(0x9201 bytes) 2024-09-16T13:29:48.115 INFO:teuthology.orchestra.run.smithi076.stdout:5675 read 0x4739166 thru 0x473ff88 (0x6e23 bytes) 2024-09-16T13:29:48.118 INFO:teuthology.orchestra.run.smithi076.stdout:5676 trunc from 0xedda119 to 0xc1bbf45 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5678 read 0x13ca1a0 thru 0x13d8c04 (0xea65 bytes) 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5679 read 0x966dad2 thru 0x96751c3 (0x76f2 bytes) 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5683 read 0x74f4024 thru 0x74f4b8c (0xb69 bytes) 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5685 read 0x217ca4 thru 0x2188d2 (0xc2f bytes) 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5686 read 0x502f44e thru 0x5031021 (0x1bd4 bytes) 2024-09-16T13:29:48.124 INFO:teuthology.orchestra.run.smithi076.stdout:5687 read 0x922cd2c thru 0x9230dc8 (0x409d bytes) 2024-09-16T13:29:48.125 INFO:teuthology.orchestra.run.smithi076.stdout:5695 writesame 0x686d7e8 thru 0x6874575 data_size 0x2(0x6d8e bytes) 2024-09-16T13:29:48.138 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:48.136+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:48.142 INFO:teuthology.orchestra.run.smithi076.stdout:5696 read 0xa9ada9b thru 0xa9bd969 (0xfecf bytes) 2024-09-16T13:29:48.145 INFO:teuthology.orchestra.run.smithi076.stdout:5697 trunc from 0xc1bbf45 to 0xe84483b 2024-09-16T13:29:48.151 INFO:teuthology.orchestra.run.smithi076.stdout:5698 write 0x1c78e7b thru 0x1c87108 (0xe28e bytes) 2024-09-16T13:29:48.170 INFO:teuthology.orchestra.run.smithi076.stdout:5699 trunc from 0xe84483b to 0x516126f 2024-09-16T13:29:48.179 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.180 INFO:teuthology.orchestra.run.smithi076.stdout:5701 read 0x2e66070 thru 0x2e6d552 (0x74e3 bytes) 2024-09-16T13:29:48.180 INFO:teuthology.orchestra.run.smithi076.stdout:5702 punch from 0x2b66567 to 0x2b6dcb9, (0x7752 bytes) 2024-09-16T13:29:48.194 INFO:teuthology.orchestra.run.smithi076.stdout:5705 write 0xd142edf thru 0xd14766f (0x4791 bytes) 2024-09-16T13:29:48.200 INFO:teuthology.orchestra.run.smithi076.stdout:5706 trunc from 0xd147670 to 0xe78c2fb 2024-09-16T13:29:48.205 INFO:teuthology.orchestra.run.smithi076.stdout:5707 punch from 0x8667927 to 0x866793b, (0x14 bytes) 2024-09-16T13:29:48.205 INFO:teuthology.orchestra.run.smithi076.stdout:5708 trunc from 0xe78c2fb to 0xe7975e2 2024-09-16T13:29:48.209 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.209 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.216 INFO:teuthology.orchestra.run.smithi076.stdout:5709 trunc from 0xe7975e2 to 0x8ec6578 2024-09-16T13:29:48.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:48.229 INFO:teuthology.orchestra.run.smithi076.stdout:5710 read 0x263be1c thru 0x2649da5 (0xdf8a bytes) 2024-09-16T13:29:48.229 INFO:teuthology.orchestra.run.smithi076.stdout:5712 punch from 0x487b8c5 to 0x4885583, (0x9cbe bytes) 2024-09-16T13:29:48.229 INFO:teuthology.orchestra.run.smithi076.stdout:5715 read 0x6117702 thru 0x6120c16 (0x9515 bytes) 2024-09-16T13:29:48.229 INFO:teuthology.orchestra.run.smithi076.stdout:5716 punch from 0x4f9b31a to 0x4fa9155, (0xde3b bytes) 2024-09-16T13:29:48.236 INFO:teuthology.orchestra.run.smithi076.stdout:5718 trunc from 0x8ec6578 to 0xedb5c4c 2024-09-16T13:29:48.239 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:48.237+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:48.241 INFO:teuthology.orchestra.run.smithi076.stdout:5719 write 0x531ef68 thru 0x53239d3 (0x4a6c bytes) 2024-09-16T13:29:48.245 INFO:teuthology.orchestra.run.smithi076.stdout:5720 read 0xa1f396d thru 0xa202d44 (0xf3d8 bytes) 2024-09-16T13:29:48.246 INFO:teuthology.orchestra.run.smithi076.stdout:5721 write 0x11111fd thru 0x11144e1 (0x32e5 bytes) 2024-09-16T13:29:48.249 INFO:teuthology.orchestra.run.smithi076.stdout:5722 punch from 0xaaa29e5 to 0xaaad0f5, (0xa710 bytes) 2024-09-16T13:29:48.249 INFO:teuthology.orchestra.run.smithi076.stdout:5726 writesame 0x872880b thru 0x872cd41 data_size 0x1d(0x4537 bytes) 2024-09-16T13:29:48.251 INFO:teuthology.orchestra.run.smithi076.stdout:5727 write 0xda00db5 thru 0xda0d361 (0xc5ad bytes) 2024-09-16T13:29:48.275 INFO:teuthology.orchestra.run.smithi076.stdout:5728 trunc from 0xedb5c4c to 0x201579a 2024-09-16T13:29:48.289 INFO:teuthology.orchestra.run.smithi076.stdout:5729 read 0x12b43de thru 0x12bdc4a (0x986d bytes) 2024-09-16T13:29:48.289 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.307 INFO:teuthology.orchestra.run.smithi076.stdout:5732 write 0xbeddfab thru 0xbedf0bf (0x1115 bytes) 2024-09-16T13:29:48.310 INFO:teuthology.orchestra.run.smithi076.stdout:5733 write 0x8815814 thru 0x882240e (0xcbfb bytes) 2024-09-16T13:29:48.325 INFO:teuthology.orchestra.run.smithi076.stdout:5734 write 0xdf91e2a thru 0xdf96361 (0x4538 bytes) 2024-09-16T13:29:48.329 INFO:teuthology.orchestra.run.smithi076.stdout:5735 read 0xde04c8 thru 0xdeec06 (0xe73f bytes) 2024-09-16T13:29:48.329 INFO:teuthology.orchestra.run.smithi076.stdout:5736 write 0x2bace11 thru 0x2bb0adb (0x3ccb bytes) 2024-09-16T13:29:48.336 INFO:teuthology.orchestra.run.smithi076.stdout:5737 writesame 0x4a4c1ca thru 0x4a4fe42 data_size 0x1(0x3c79 bytes) 2024-09-16T13:29:48.339 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:48.338+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:48.345 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.345 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:48.355 INFO:teuthology.orchestra.run.smithi076.stdout:5738 read 0x1da7842 thru 0x1dae9cc (0x718b bytes) 2024-09-16T13:29:48.355 INFO:teuthology.orchestra.run.smithi076.stdout:5739 read 0xb03c52d thru 0xb049b4b (0xd61f bytes) 2024-09-16T13:29:48.355 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:5743 write 0xec03d37 thru 0xec087d9 (0x4aa3 bytes) 2024-09-16T13:29:48.381 INFO:teuthology.orchestra.run.smithi076.stdout:5745 trunc from 0xec087da to 0x17cc0d3 2024-09-16T13:29:48.396 INFO:teuthology.orchestra.run.smithi076.stdout:5747 read 0x199459 thru 0x1a10bc (0x7c64 bytes) 2024-09-16T13:29:48.396 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.408 INFO:teuthology.orchestra.run.smithi076.stdout:5749 trunc from 0x17cc0d3 to 0xa6ba1ff 2024-09-16T13:29:48.414 INFO:teuthology.orchestra.run.smithi076.stdout:5750 punch from 0x2e848bb to 0x2e8622b, (0x1970 bytes) 2024-09-16T13:29:48.414 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.414 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.414 INFO:teuthology.orchestra.run.smithi076.stdout:5753 read 0x3ab180c thru 0x3abb1b8 (0x99ad bytes) 2024-09-16T13:29:48.414 INFO:teuthology.orchestra.run.smithi076.stdout:5754 trunc from 0xa6ba1ff to 0xb21812a 2024-09-16T13:29:48.420 INFO:teuthology.orchestra.run.smithi076.stdout:5755 write 0x70ca0a4 thru 0x70d3f3e (0x9e9b bytes) 2024-09-16T13:29:48.424 INFO:teuthology.orchestra.run.smithi076.stdout:5756 read 0x25b2f51 thru 0x25c218b (0xf23b bytes) 2024-09-16T13:29:48.429 INFO:teuthology.orchestra.run.smithi076.stdout:5757 trunc from 0xb21812a to 0x7163d69 2024-09-16T13:29:48.435 INFO:teuthology.orchestra.run.smithi076.stdout:5759 read 0x50c0ef0 thru 0x50c5266 (0x4377 bytes) 2024-09-16T13:29:48.435 INFO:teuthology.orchestra.run.smithi076.stdout:5760 punch from 0x51f830f to 0x5207a9e, (0xf78f bytes) 2024-09-16T13:29:48.438 INFO:teuthology.orchestra.run.smithi076.stdout:5762 trunc from 0x7163d69 to 0x9a37fea 2024-09-16T13:29:48.440 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:48.438+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:48.443 INFO:teuthology.orchestra.run.smithi076.stdout:5763 writesame 0x291f6ab thru 0x29255bd data_size 0x3d(0x5f13 bytes) 2024-09-16T13:29:48.447 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.447 INFO:teuthology.orchestra.run.smithi076.stdout:5766 write 0x40c08cd thru 0x40c9be1 (0x9315 bytes) 2024-09-16T13:29:48.452 INFO:teuthology.orchestra.run.smithi076.stdout:5767 read 0x1a692c1 thru 0x1a73869 (0xa5a9 bytes) 2024-09-16T13:29:48.456 INFO:teuthology.orchestra.run.smithi076.stdout:5768 trunc from 0x9a37fea to 0x694ade8 2024-09-16T13:29:48.464 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.470 INFO:teuthology.orchestra.run.smithi076.stdout:5770 trunc from 0x694ade8 to 0x1a0f1c7 2024-09-16T13:29:48.476 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.476 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.481 INFO:teuthology.orchestra.run.smithi076.stdout:5771 read 0x12a5e3e thru 0x12b3c57 (0xde1a bytes) 2024-09-16T13:29:48.481 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:48.493 INFO:teuthology.orchestra.run.smithi076.stdout:5774 trunc from 0x1a0f1c7 to 0xb876f60 2024-09-16T13:29:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:5779 read 0x3cc194c thru 0x3cd1820 (0xfed5 bytes) 2024-09-16T13:29:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:5780 writesame 0x504444b thru 0x504d430 data_size 0x2(0x8fe6 bytes) 2024-09-16T13:29:48.515 INFO:teuthology.orchestra.run.smithi076.stdout:5781 writesame 0xcb759af thru 0xcb82e1e data_size 0xb0(0xd470 bytes) 2024-09-16T13:29:48.515 INFO:teuthology.orchestra.run.smithi076.stdout:5783 punch from 0x3bfac3e to 0x3bfc00d, (0x13cf bytes) 2024-09-16T13:29:48.517 INFO:teuthology.orchestra.run.smithi076.stdout:5784 trunc from 0xcb82e1f to 0xeb9afb3 2024-09-16T13:29:48.523 INFO:teuthology.orchestra.run.smithi076.stdout:5785 write 0x118a82 thru 0x11f080 (0x65ff bytes) 2024-09-16T13:29:48.524 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:48.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:48.525 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-16T13:29:48.525 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:29:48.525 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:29:48.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-16T13:29:48.529 INFO:teuthology.orchestra.run.smithi076.stdout:5786 writesame 0xb6ab9b7 thru 0xb6afc24 data_size 0xb(0x426e bytes) 2024-09-16T13:29:48.534 INFO:teuthology.orchestra.run.smithi076.stdout:5787 read 0x67cb17c thru 0x67d7aff (0xc984 bytes) 2024-09-16T13:29:48.534 INFO:teuthology.orchestra.run.smithi076.stdout:5789 read 0x983c8d2 thru 0x9840bf1 (0x4320 bytes) 2024-09-16T13:29:48.534 INFO:teuthology.orchestra.run.smithi076.stdout:5790 writesame 0xc6f330e thru 0xc6f6e58 data_size 0x1(0x3b4b bytes) 2024-09-16T13:29:48.541 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:48.539+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:48.553 INFO:teuthology.orchestra.run.smithi076.stdout:5791 read 0x42fed03 thru 0x43056c3 (0x69c1 bytes) 2024-09-16T13:29:48.553 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.553 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.553 INFO:teuthology.orchestra.run.smithi076.stdout:5795 write 0x6ba285e thru 0x6bb15d8 (0xed7b bytes) 2024-09-16T13:29:48.558 INFO:teuthology.orchestra.run.smithi076.stdout:5797 write 0xcc11528 thru 0xcc132e0 (0x1db9 bytes) 2024-09-16T13:29:48.561 INFO:teuthology.orchestra.run.smithi076.stdout:5798 read 0x6cf0d89 thru 0x6cf89c8 (0x7c40 bytes) 2024-09-16T13:29:48.561 INFO:teuthology.orchestra.run.smithi076.stdout:5799 read 0xc775153 thru 0xc77d20c (0x80ba bytes) 2024-09-16T13:29:48.561 INFO:teuthology.orchestra.run.smithi076.stdout:5800 write 0x33c6b6b thru 0x33d4c2e (0xe0c4 bytes) 2024-09-16T13:29:48.566 INFO:teuthology.orchestra.run.smithi076.stdout:5801 write 0xa324006 thru 0xa32dc4e (0x9c49 bytes) 2024-09-16T13:29:48.572 INFO:teuthology.orchestra.run.smithi076.stdout:5804 read 0x4ebf43e thru 0x4ec1415 (0x1fd8 bytes) 2024-09-16T13:29:48.572 INFO:teuthology.orchestra.run.smithi076.stdout:5805 read 0x77dcc20 thru 0x77de003 (0x13e4 bytes) 2024-09-16T13:29:48.572 INFO:teuthology.orchestra.run.smithi076.stdout:5806 write 0xe85fddc thru 0xe869940 (0x9b65 bytes) 2024-09-16T13:29:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:5807 punch from 0x9f3282c to 0x9f3cdc0, (0xa594 bytes) 2024-09-16T13:29:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:5808 write 0xc491c2e thru 0xc4a1ac4 (0xfe97 bytes) 2024-09-16T13:29:48.581 INFO:teuthology.orchestra.run.smithi076.stdout:5809 write 0x2639a30 thru 0x264399d (0x9f6e bytes) 2024-09-16T13:29:48.586 INFO:teuthology.orchestra.run.smithi076.stdout:5811 write 0xe3e7f7d thru 0xe3e8e47 (0xecb bytes) 2024-09-16T13:29:48.592 INFO:teuthology.orchestra.run.smithi076.stdout:5814 writesame 0x5cc2524 thru 0x5ccd8c6 data_size 0x1(0xb3a3 bytes) 2024-09-16T13:29:48.601 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.601 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:48.635 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:48.635 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:29:48.635 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:48.638 INFO:teuthology.orchestra.run.smithi076.stdout:5816 read 0x19ed402 thru 0x19fbd1f (0xe91e bytes) 2024-09-16T13:29:48.639 INFO:teuthology.orchestra.run.smithi076.stdout:5817 writesame 0x1076b30 thru 0x107a290 data_size 0x1(0x3761 bytes) 2024-09-16T13:29:48.661 INFO:teuthology.orchestra.run.smithi076.stdout:5818 write 0x5eb4c5f thru 0x5eba373 (0x5715 bytes) 2024-09-16T13:29:48.661 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:48.659+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:48.666 INFO:teuthology.orchestra.run.smithi076.stdout:5819 write 0x8511e03 thru 0x851ca2d (0xac2b bytes) 2024-09-16T13:29:48.672 INFO:teuthology.orchestra.run.smithi076.stdout:5822 write 0x923523a thru 0x92446e0 (0xf4a7 bytes) 2024-09-16T13:29:48.677 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.693 INFO:teuthology.orchestra.run.smithi076.stdout:5824 trunc from 0xeb9afb3 to 0x16788a4 2024-09-16T13:29:48.722 INFO:teuthology.orchestra.run.smithi076.stdout:5827 writesame 0x3166b12 thru 0x31756df data_size 0x3(0xebce bytes) 2024-09-16T13:29:48.723 INFO:teuthology.orchestra.run.smithi076.stdout:5828 trunc from 0x31756e0 to 0x285fc56 2024-09-16T13:29:48.743 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:48.741+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:48.747 INFO:teuthology.orchestra.run.smithi076.stdout:5830 writesame 0xe484825 thru 0xe493fd1 data_size 0xf(0xf7ad bytes) 2024-09-16T13:29:48.748 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.749 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:48.762 INFO:teuthology.orchestra.run.smithi076.stdout:5831 trunc from 0xe493fd2 to 0x24ab789 2024-09-16T13:29:48.767 INFO:teuthology.orchestra.run.smithi076.stdout:5832 write 0xef54e0 thru 0xf010f7 (0xbc18 bytes) 2024-09-16T13:29:48.783 INFO:teuthology.orchestra.run.smithi076.stdout:5833 writesame 0x6535b6f thru 0x653f85e data_size 0x30(0x9cf0 bytes) 2024-09-16T13:29:48.783 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.783 INFO:teuthology.orchestra.run.smithi076.stdout:5840 write 0x41401fc thru 0x4145281 (0x5086 bytes) 2024-09-16T13:29:48.786 INFO:teuthology.orchestra.run.smithi076.stdout:5841 read 0x3b24f89 thru 0x3b300ce (0xb146 bytes) 2024-09-16T13:29:48.793 INFO:teuthology.orchestra.run.smithi076.stdout:5842 write 0x78e8538 thru 0x78f0cb1 (0x877a bytes) 2024-09-16T13:29:48.797 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.809 INFO:teuthology.orchestra.run.smithi076.stdout:5845 write 0xd439da0 thru 0xd43c6ae (0x290f bytes) 2024-09-16T13:29:48.812 INFO:teuthology.orchestra.run.smithi076.stdout:5846 punch from 0x8ad8615 to 0x8add529, (0x4f14 bytes) 2024-09-16T13:29:48.825 INFO:teuthology.orchestra.run.smithi076.stdout:5848 trunc from 0xd43c6af to 0x296afbe 2024-09-16T13:29:48.835 INFO:teuthology.orchestra.run.smithi076.stdout:5849 punch from 0x28d44dc to 0x28d89e5, (0x4509 bytes) 2024-09-16T13:29:48.835 INFO:teuthology.orchestra.run.smithi076.stdout:5850 punch from 0x1400bc7 to 0x1402930, (0x1d69 bytes) 2024-09-16T13:29:48.836 INFO:teuthology.orchestra.run.smithi076.stdout:5851 trunc from 0x296afbe to 0x3cc607b 2024-09-16T13:29:48.841 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.841 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.843 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:48.842+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:48.845 INFO:teuthology.orchestra.run.smithi076.stdout:5857 trunc from 0x3cc607b to 0x73d592 2024-09-16T13:29:48.867 INFO:teuthology.orchestra.run.smithi076.stdout:5858 read 0xeffb8 thru 0xfea30 (0xea79 bytes) 2024-09-16T13:29:48.869 INFO:teuthology.orchestra.run.smithi076.stdout:5859 read 0x2cf52f thru 0x2d9472 (0x9f44 bytes) 2024-09-16T13:29:48.869 INFO:teuthology.orchestra.run.smithi076.stdout:5860 read 0x1c2daa thru 0x1d1233 (0xe48a bytes) 2024-09-16T13:29:48.869 INFO:teuthology.orchestra.run.smithi076.stdout:5861 read 0x623b23 thru 0x631cf5 (0xe1d3 bytes) 2024-09-16T13:29:48.870 INFO:teuthology.orchestra.run.smithi076.stdout:5864 read 0x2cc4b2 thru 0x2cd3b5 (0xf04 bytes) 2024-09-16T13:29:48.870 INFO:teuthology.orchestra.run.smithi076.stdout:5866 read 0x537394 thru 0x541c1b (0xa888 bytes) 2024-09-16T13:29:48.877 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.877 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:48.891 INFO:teuthology.orchestra.run.smithi076.stdout:5867 write 0xdf5481b thru 0xdf5e844 (0xa02a bytes) 2024-09-16T13:29:48.896 INFO:teuthology.orchestra.run.smithi076.stdout:5868 writesame 0x31318ca thru 0x313ca1c data_size 0x7(0xb153 bytes) 2024-09-16T13:29:48.901 INFO:teuthology.orchestra.run.smithi076.stdout:5869 writesame 0x5ec183b thru 0x5eca03d data_size 0x1(0x8803 bytes) 2024-09-16T13:29:48.932 INFO:teuthology.orchestra.run.smithi076.stdout:5870 read 0xaa99817 thru 0xaa9cd44 (0x352e bytes) 2024-09-16T13:29:48.933 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.933 INFO:teuthology.orchestra.run.smithi076.stdout:5874 read 0x6ee00c9 thru 0x6ee617a (0x60b2 bytes) 2024-09-16T13:29:48.933 INFO:teuthology.orchestra.run.smithi076.stdout:5877 write 0xd874c88 thru 0xd87dcd6 (0x904f bytes) 2024-09-16T13:29:48.937 INFO:teuthology.orchestra.run.smithi076.stdout:5878 write 0x12463bf thru 0x1255032 (0xec74 bytes) 2024-09-16T13:29:48.944 INFO:teuthology.orchestra.run.smithi076.stdout:5879 read 0xd91fdc7 thru 0xd9274b5 (0x76ef bytes) 2024-09-16T13:29:48.944 INFO:teuthology.orchestra.run.smithi076.stdout:5880 punch from 0xc4c04c1 to 0xc4c5b29, (0x5668 bytes) 2024-09-16T13:29:48.944 INFO:teuthology.orchestra.run.smithi076.stdout:5881 writesame 0x7b04deb thru 0x7b096ca data_size 0x58(0x48e0 bytes) 2024-09-16T13:29:48.944 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:48.942+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:48.948 INFO:teuthology.orchestra.run.smithi076.stdout:5882 write 0x92b7ee0 thru 0x92c5a45 (0xdb66 bytes) 2024-09-16T13:29:48.955 INFO:teuthology.orchestra.run.smithi076.stdout:5883 read 0xbbf22c9 thru 0xbbfe722 (0xc45a bytes) 2024-09-16T13:29:48.955 INFO:teuthology.orchestra.run.smithi076.stdout:5884 read 0x8827f6c thru 0x8829102 (0x1197 bytes) 2024-09-16T13:29:48.955 INFO:teuthology.orchestra.run.smithi076.stdout:5885 write 0xa699a1a thru 0xa6a4cfd (0xb2e4 bytes) 2024-09-16T13:29:48.960 INFO:teuthology.orchestra.run.smithi076.stdout:5886 read 0xaf60712 thru 0xaf690ef (0x89de bytes) 2024-09-16T13:29:48.960 INFO:teuthology.orchestra.run.smithi076.stdout:5888 write 0xb42755d thru 0xb436e4b (0xf8ef bytes) 2024-09-16T13:29:48.967 INFO:teuthology.orchestra.run.smithi076.stdout:5889 read 0x1ad6ac1 thru 0x1ae38dc (0xce1c bytes) 2024-09-16T13:29:48.967 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.967 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.967 INFO:teuthology.orchestra.run.smithi076.stdout:5892 read 0x626fd1d thru 0x627eeba (0xf19e bytes) 2024-09-16T13:29:48.973 INFO:teuthology.orchestra.run.smithi076.stdout:5893 trunc from 0xdf5e845 to 0x8f150af 2024-09-16T13:29:48.984 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.984 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.985 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:48.985 INFO:teuthology.orchestra.run.smithi076.stdout:5904 punch from 0x197f42a to 0x198717a, (0x7d50 bytes) 2024-09-16T13:29:48.985 INFO:teuthology.orchestra.run.smithi076.stdout:5905 write 0x47ac7bb thru 0x47b767a (0xaec0 bytes) 2024-09-16T13:29:48.991 INFO:teuthology.orchestra.run.smithi076.stdout:5906 writesame 0x107fa70 thru 0x1083c42 data_size 0x29(0x41d3 bytes) 2024-09-16T13:29:48.996 INFO:teuthology.orchestra.run.smithi076.stdout:5907 read 0x8f06033 thru 0x8f0d4b2 (0x7480 bytes) 2024-09-16T13:29:48.996 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:48.997 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:48.997 INFO:teuthology.orchestra.run.smithi076.stdout:5908 write 0x5ad0c55 thru 0x5ae04f0 (0xf89c bytes) 2024-09-16T13:29:49.003 INFO:teuthology.orchestra.run.smithi076.stdout:5909 read 0x7bdcc98 thru 0x7bebe28 (0xf191 bytes) 2024-09-16T13:29:49.003 INFO:teuthology.orchestra.run.smithi076.stdout:5911 writesame 0x668510a thru 0x668df03 data_size 0x11(0x8dfa bytes) 2024-09-16T13:29:49.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:49.008 INFO:teuthology.orchestra.run.smithi076.stdout:5912 read 0x6157547 thru 0x616237a (0xae34 bytes) 2024-09-16T13:29:49.008 INFO:teuthology.orchestra.run.smithi076.stdout:5914 write 0x3a4cea2 thru 0x3a56094 (0x91f3 bytes) 2024-09-16T13:29:49.013 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.013 INFO:teuthology.orchestra.run.smithi076.stdout:5919 read 0x1ec9115 thru 0x1ecc656 (0x3542 bytes) 2024-09-16T13:29:49.021 INFO:teuthology.orchestra.run.smithi076.stdout:5920 write 0xb491cb1 thru 0xb4a105d (0xf3ad bytes) 2024-09-16T13:29:49.027 INFO:teuthology.orchestra.run.smithi076.stdout:5922 read 0x887aa72 thru 0x8889ea1 (0xf430 bytes) 2024-09-16T13:29:49.027 INFO:teuthology.orchestra.run.smithi076.stdout:5923 read 0x6477cdb thru 0x647a41a (0x2740 bytes) 2024-09-16T13:29:49.027 INFO:teuthology.orchestra.run.smithi076.stdout:5924 read 0x1274206 thru 0x12785e7 (0x43e2 bytes) 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:5925 read 0x23bdeca thru 0x23c1f4e (0x4085 bytes) 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:5926 punch from 0xa0961f to 0xa0cb55, (0x3536 bytes) 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.028 INFO:teuthology.orchestra.run.smithi076.stdout:5930 writesame 0x29ed6e4 thru 0x29f6001 data_size 0x1(0x891e bytes) 2024-09-16T13:29:49.045 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:49.044+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:49.078 INFO:teuthology.orchestra.run.smithi076.stdout:5932 writesame 0x646ce88 thru 0x646f4b1 data_size 0x5(0x262a bytes) 2024-09-16T13:29:49.082 INFO:teuthology.orchestra.run.smithi076.stdout:5933 read 0xa2109a1 thru 0xa215d3b (0x539b bytes) 2024-09-16T13:29:49.083 INFO:teuthology.orchestra.run.smithi076.stdout:5934 read 0x90cb5cf thru 0x90d7bbd (0xc5ef bytes) 2024-09-16T13:29:49.091 INFO:teuthology.orchestra.run.smithi076.stdout:5936 write 0xd623fc3 thru 0xd631013 (0xd051 bytes) 2024-09-16T13:29:49.097 INFO:teuthology.orchestra.run.smithi076.stdout:5938 writesame 0x62ad6df thru 0x62b47e1 data_size 0x1(0x7103 bytes) 2024-09-16T13:29:49.115 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.115 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.124 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:49.127 INFO:teuthology.orchestra.run.smithi076.stdout:5940 punch from 0x6a25f85 to 0x6a2b925, (0x59a0 bytes) 2024-09-16T13:29:49.127 INFO:teuthology.orchestra.run.smithi076.stdout:5943 write 0x513c747 thru 0x5147781 (0xb03b bytes) 2024-09-16T13:29:49.138 INFO:teuthology.orchestra.run.smithi076.stdout:5944 write 0xe248bd1 thru 0xe2549f4 (0xbe24 bytes) 2024-09-16T13:29:49.142 INFO:teuthology.orchestra.run.smithi076.stdout:5945 writesame 0x3ff28b4 thru 0x4002422 data_size 0x1(0xfb6f bytes) 2024-09-16T13:29:49.146 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:49.145+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:49.189 INFO:teuthology.orchestra.run.smithi076.stdout:5946 writesame 0x59eca10 thru 0x59f1270 data_size 0x1(0x4861 bytes) 2024-09-16T13:29:49.204 INFO:teuthology.orchestra.run.smithi076.stdout:5947 punch from 0x77a558b to 0x77a5e3c, (0x8b1 bytes) 2024-09-16T13:29:49.204 INFO:teuthology.orchestra.run.smithi076.stdout:5948 read 0x95ee941 thru 0x95f23b9 (0x3a79 bytes) 2024-09-16T13:29:49.204 INFO:teuthology.orchestra.run.smithi076.stdout:5949 read 0x70e0d1a thru 0x70e1257 (0x53e bytes) 2024-09-16T13:29:49.205 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.205 INFO:teuthology.orchestra.run.smithi076.stdout:5951 read 0x64e758c thru 0x64ef755 (0x81ca bytes) 2024-09-16T13:29:49.205 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.205 INFO:teuthology.orchestra.run.smithi076.stdout:5954 write 0x203bc25 thru 0x204509b (0x9477 bytes) 2024-09-16T13:29:49.210 INFO:teuthology.orchestra.run.smithi076.stdout:5956 trunc from 0xe2549f5 to 0xc57ae9c 2024-09-16T13:29:49.221 INFO:teuthology.orchestra.run.smithi076.stdout:5959 writesame 0x2548da3 thru 0x254ca23 data_size 0x3(0x3c81 bytes) 2024-09-16T13:29:49.227 INFO:teuthology.orchestra.run.smithi076.stdout:5961 writesame 0x6cb0558 thru 0x6cb7160 data_size 0x15(0x6c09 bytes) 2024-09-16T13:29:49.231 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.231 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.231 INFO:teuthology.orchestra.run.smithi076.stdout:5962 read 0x8a5a7dd thru 0x8a622ff (0x7b23 bytes) 2024-09-16T13:29:49.231 INFO:teuthology.orchestra.run.smithi076.stdout:5964 writesame 0x49ff24d thru 0x4a04ad0 data_size 0x67(0x5884 bytes) 2024-09-16T13:29:49.237 INFO:teuthology.orchestra.run.smithi076.stdout:5965 trunc from 0xc57ae9c to 0xd1251e1 2024-09-16T13:29:49.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:49.242 INFO:teuthology.orchestra.run.smithi076.stdout:5966 write 0x1e0bcd thru 0x1e4d10 (0x4144 bytes) 2024-09-16T13:29:49.246 INFO:teuthology.orchestra.run.smithi076.stdout:5967 punch from 0x8c5cbfb to 0x8c5e2b3, (0x16b8 bytes) 2024-09-16T13:29:49.246 INFO:teuthology.orchestra.run.smithi076.stdout:5968 read 0x6759301 thru 0x675d340 (0x4040 bytes) 2024-09-16T13:29:49.246 INFO:teuthology.orchestra.run.smithi076.stdout:5970 punch from 0xb4b5a72 to 0xb4bf751, (0x9cdf bytes) 2024-09-16T13:29:49.247 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:49.246+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:49.253 INFO:teuthology.orchestra.run.smithi076.stdout:5971 write 0xe69a779 thru 0xe69b2ae (0xb36 bytes) 2024-09-16T13:29:49.257 INFO:teuthology.orchestra.run.smithi076.stdout:5972 writesame 0x36f7caa thru 0x36f922a data_size 0x1(0x1581 bytes) 2024-09-16T13:29:49.264 INFO:teuthology.orchestra.run.smithi076.stdout:5973 writesame 0xe589dd0 thru 0xe58eaf9 data_size 0x11(0x4d2a bytes) 2024-09-16T13:29:49.267 INFO:teuthology.orchestra.run.smithi076.stdout:5974 write 0xae00f81 thru 0xae0d1ca (0xc24a bytes) 2024-09-16T13:29:49.274 INFO:teuthology.orchestra.run.smithi076.stdout:5975 punch from 0x2c1e849 to 0x2c2787a, (0x9031 bytes) 2024-09-16T13:29:49.274 INFO:teuthology.orchestra.run.smithi076.stdout:5976 writesame 0x657d4c6 thru 0x6584fc4 data_size 0x17(0x7aff bytes) 2024-09-16T13:29:49.279 INFO:teuthology.orchestra.run.smithi076.stdout:5977 read 0x811563 thru 0x81b67f (0xa11d bytes) 2024-09-16T13:29:49.279 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.289 INFO:teuthology.orchestra.run.smithi076.stdout:5980 trunc from 0xe69b2af to 0x6d283bd 2024-09-16T13:29:49.301 INFO:teuthology.orchestra.run.smithi076.stdout:5982 write 0x6b4f40e thru 0x6b525ff (0x31f2 bytes) 2024-09-16T13:29:49.311 INFO:teuthology.orchestra.run.smithi076.stdout:5983 trunc from 0x6d283bd to 0x2afa18b 2024-09-16T13:29:49.332 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.333 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.344 INFO:teuthology.orchestra.run.smithi076.stdout:5988 writesame 0x6fa426b thru 0x6fb22d6 data_size 0x6a(0xe06c bytes) 2024-09-16T13:29:49.344 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.344 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.344 INFO:teuthology.orchestra.run.smithi076.stdout:5991 clone 44 order 19 su 4096 sc 12 2024-09-16T13:29:49.348 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:49.347+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:49.360 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.360 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:49.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:29:49.453 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:49.451+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:49.494 INFO:teuthology.orchestra.run.smithi076.stdout:truncating image image_client.0-clone43 from 0x6fb22d7 (overlap 0x73d592) to 0x684036 2024-09-16T13:29:49.526 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.526 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:49.550 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:49.549+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:49.557 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #42, image image_client.0-clone42 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent43 2024-09-16T13:29:49.651 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:49.650+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:49.672 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.672 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:49.765 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:49.763+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:49.798 INFO:teuthology.orchestra.run.smithi076.stdout:5992 write 0x8ecca7 thru 0x8f9e33 (0xd18d bytes) 2024-09-16T13:29:49.820 INFO:teuthology.orchestra.run.smithi076.stdout:5993 punch from 0x19a3528 to 0x19af663, (0xc13b bytes) 2024-09-16T13:29:49.820 INFO:teuthology.orchestra.run.smithi076.stdout:5995 punch from 0x1b542c7 to 0x1b5dbf2, (0x992b bytes) 2024-09-16T13:29:49.820 INFO:teuthology.orchestra.run.smithi076.stdout:5996 writesame 0x6ace15 thru 0x6ae80d data_size 0x1(0x19f9 bytes) 2024-09-16T13:29:49.831 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.831 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.834 INFO:teuthology.orchestra.run.smithi076.stdout:5997 writesame 0xd329f42 thru 0xd32f3f0 data_size 0x13(0x54af bytes) 2024-09-16T13:29:49.834 INFO:teuthology.orchestra.run.smithi076.stdout:5998 punch from 0x1377194 to 0x1383ca6, (0xcb12 bytes) 2024-09-16T13:29:49.835 INFO:teuthology.orchestra.run.smithi076.stdout:compare and write disabled 2024-09-16T13:29:49.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:49.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:49.852 INFO:teuthology.orchestra.run.smithi076.stdout:checking clone #43, image image_client.0-clone43 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent44 2024-09-16T13:29:49.886 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:49.884+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:49.944 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:49.944 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:49.953 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:49.951+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:49.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:49.979 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:49.979 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:49.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:50.012 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:50.012 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:50.027 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:50.028 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:50.043 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:50.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:50.054 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:50.053+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:50.059 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:50.076 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:50.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:50.092 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:50.095 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:50.095 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:50.095 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:50.096 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:50.096 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:50.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:50.189 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:50.187+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:50.245 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.532863711999994, 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:29:50.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:29:50.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:50.252 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.523324600999999, 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:29:50.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:50.256 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.556867865999997, 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:50.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:29:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.262 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.846456414999999, 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:29:50.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:29:50.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:50.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.269 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.835809365999999, 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.273 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 50.168747785000001, 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:29:50.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:29:50.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:50.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:29:50.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:29:50.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:29:50.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.601116729000001, 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:29:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:29:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:29:50.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:29:50.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:50.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:29:50.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:29:50.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:29:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:29:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.583983835000002, 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:29:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:29:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.623594889, 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:29:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:29:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:29:50.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.607106168000001, 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:29:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:29:50.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:50.311 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.227221131, 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.315 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.226628202000001, 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.316 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:29:50.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:29:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.226613538999999, 2024-09-16T13:29:50.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:29:50.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.324 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.932036887000001, 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:29:50.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.918755661000001, 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:29:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:29:50.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.334 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9535059860000001, 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.335 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:29:50.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:29:50.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.9372292570000003, 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:29:50.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.6658843430000001, 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:29:50.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:29:50.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.350 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.6656894480000002, 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:29:50.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:29:50.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:50.356 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.6464265300000003, 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:29:50.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:29:50.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.360 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:50.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:50.362 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:50.359+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:50.436 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:29:50.462 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:50.460+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:50.508 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:50.508 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:50.508 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:50.508 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:50.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:50.563 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:50.560+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:50.656 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:50.656 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:50.656 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:50.656 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:50.656 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:50.664 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:50.661+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:50.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:50.764 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:50.764 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:50.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:50.765 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:50.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.765 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:50.765 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 76.031201631000002, 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:29:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:29:50.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.588310308999993, 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.773 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:29:50.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 64.074852281000005, 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:29:50.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:50.783 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 64.064763013999993, 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:29:50.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:29:50.787 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.786049986999998, 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:29:50.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:29:50.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:29:50.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.794 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.837891014, 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:29:50.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:29:50.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:29:50.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.801 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750711029000001, 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:29:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:29:50.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750693430999998, 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.806 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.810 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750678732000001, 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:29:50.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:29:50.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750667800999999, 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.815 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.819 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750652182, 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750630918999999, 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.825 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:29:50.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:50.829 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750615909, 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:50.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:29:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750599827999999, 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:29:50.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750559961, 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:29:50.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:29:50.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.750548608999999, 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.845 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:50.849 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.0259977229999997, 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:50.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.0247093970000005, 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:29:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.0244374619999999, 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:50.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:29:50.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.0189588399999998, 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:29:50.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:29:50.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:29:50.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:50.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:50.870 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:50.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:50.872 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:50.869+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:50.972 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:50.970+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:51.011 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.011 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:51.011 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:51.011 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:51.104 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:51.102+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:51.159 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:51.159 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:51.159 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:51.159 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:51.209 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:51.207+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:51.274 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:51.272+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.307 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.617665355, 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:29:51.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:29:51.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.313 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.608093877000002, 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:29:51.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.760664935999998, 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:29:51.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:29:51.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014397775999996, 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:29:51.322 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:29:51.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.326 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014390035000005, 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:29:51.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:29:51.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:29:51.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014381475999997, 2024-09-16T13:29:51.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:29:51.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014372598999998, 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.336 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:29:51.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:51.340 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014358733999998, 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:29:51.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:29:51.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014350266999998, 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.345 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:29:51.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.014341463999997, 2024-09-16T13:29:51.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:29:51.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.353 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.390871406000002, 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:29:51.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:29:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.359 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.377642725000001, 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:29:51.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 49.863816536000002, 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:29:51.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:29:51.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:29:51.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:29:51.370 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 49.846760658000001, 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:29:51.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.942610078000001, 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:29:51.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:29:51.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.931971038, 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:29:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100149 4.1f 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.5724978860000007, 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037897414999999997, 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100149 2024-09-16T13:29:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732188+0000", 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8050000000000001e-06 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734548+0000", 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002359758 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734560+0000", 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1919999999999999e-05 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734569+0000", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0620000000000007e-06 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734606+0000", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6939e-05 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734726+0000", 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119768 2024-09-16T13:29:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735030+0000", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000304109 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765403+0000", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030373052000000001 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.769969+0000", 2024-09-16T13:29:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045659089999999999 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770038+0000", 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8748e-05 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770084+0000", 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6344999999999998e-05 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100148 4.12 e95/32 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head v 95'622, mlcod=92'620)", 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:29:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.5699646180000002, 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037682625999999997, 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737443+0000", 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027229519999999998 2024-09-16T13:29:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737460+0000", 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6569e-05 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737471+0000", 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1401e-05 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737528+0000", 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7046999999999998e-05 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737551+0000", 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2459e-05 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772360+0000", 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034809253999999998 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772402+0000", 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2459999999999997e-05 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100332 4.1f 4:f86ca47e:::rbd_data.11a97e9c0852.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2792831170000003, 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054467106000000001, 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100332 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025403+0000", 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6330000000000001e-06 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025408+0000", 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9049999999999996e-06 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025420+0000", 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2029e-05 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025424+0000", 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4819999999999996e-06 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025459+0000", 2024-09-16T13:29:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4489000000000001e-05 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025514+0000", 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4737000000000001e-05 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.028379+0000", 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028655379999999999 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079727+0000", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051347589999999999 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079800+0000", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3681000000000001e-05 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079840+0000", 2024-09-16T13:29:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9243999999999999e-05 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079868+0000", 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:100332 4.1f e95/32 ondisk, result = 0)", 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.2685154680000004, 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043713039000000002, 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036170+0000", 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4399999999999999e-07 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036173+0000", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036182+0000", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3689999999999996e-06 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036187+0000", 2024-09-16T13:29:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2900000000000002e-06 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079788+0000", 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043600379000000002 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079796+0000", 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9270000000000005e-06 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079882+0000", 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6323999999999995e-05 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.409 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:51.411 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:51.408+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:51.511 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:51.509+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:51.554 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:29:51.554 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:51.613 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:51.610+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:51.663 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.663 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:51.663 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:51.663 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:51.663 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:51.714 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:51.712+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 106.47904354400001, 2024-09-16T13:29:51.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:51.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:51.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.777124609, 2024-09-16T13:29:51.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:51.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.77710209999999, 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:51.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:51.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.777055773, 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:51.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:51.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.776889016, 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.797 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:51.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:51.803 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.776874251, 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.804 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:51.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:51.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.431551934, 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.810 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:51.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:51.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:51.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:51.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:51.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:51.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:51.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:51.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:51.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.821 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.431514256, 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:51.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:51.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:51.828 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.431491443, 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:51.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.43140622999999, 2024-09-16T13:29:51.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:51.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:51.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:51.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.124827304999997, 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:51.842 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:51.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.098919070999997, 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:51.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:51.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.030153916000003, 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:51.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:51.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:51.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:51.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.528611099000003, 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:51.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 28.33368578, 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:29:51.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:29:51.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 19.177808745, 2024-09-16T13:29:51.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:29:51.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:29:51.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.898856915, 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:51.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.123502208, 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:29:51.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:51.878 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.026231787, 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:29:51.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 10.024550533999999, 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:29:51.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:29:51.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:51.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:51.887 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:51.888 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:51.889 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:51.886+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:51.989 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:51.988+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:51.996 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:51.996 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:52.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:52.090 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:52.087+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:52.101 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:52.101 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:52.102 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:52.102 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:52.102 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:52.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:52.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:52.194 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:52.191+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:52.221 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:52.221 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:52.237 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:52.237 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:52.251 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.251 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.252 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:52.252 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:52.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:52.268 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:52.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:52.283 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:52.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:52.292 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:52.289+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:52.299 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:52.299 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:52.314 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:52.315 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:52.330 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:52.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:52.368 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.369 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:52.393 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:52.391+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:52.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:29:52.506 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:52.503+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:52.531 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.531 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:52.594 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:52.593+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:52.680 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.680 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:52.733 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:52.730+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:52.787 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.787 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:52.796 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:52.795+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:52.897 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:52.895+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:52.935 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:52.935 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:52.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:52.998 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:52.997+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:53.081 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.082 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:53.134 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:53.131+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:53.188 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.188 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:53.238 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:53.235+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:53.301 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:53.300+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:53.335 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.335 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:53.402 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:53.399+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:53.441 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.442 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:53.452 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:29:53.507 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:53.505+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:53.604 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:53.601+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:53.645 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:53.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:53.646 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T13:29:53.646 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-16T13:29:53.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0 2024-09-16T13:29:53.671 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.671 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:53.705 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:53.702+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:53.778 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.778 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:53.807 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:53.804+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:53.884 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.884 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:53.907 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:53.904+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:53.991 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:53.991 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:53.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:54.008 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:54.005+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:54.097 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:54.097 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:54.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:54.109 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:54.107+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:54.205 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:54.206 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:54.210 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:54.208+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:54.214 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:54.311 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:54.308+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:54.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:54.352 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:54.352 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:54.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:54.411 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:54.409+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:54.461 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:54.461 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-16T13:29:54.463 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:54.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:54.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:54.486 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:54.501 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:54.501 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:54.512 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:54.510+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:54.518 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:54.519 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:54.535 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:54.535 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:54.554 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:54.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:54.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:54.570 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:54.576 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:54.576 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:54.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:54.587 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:54.628 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:54.626+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:54.682 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:54.683 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:29:54.683 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:29:54.683 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:29:54.683 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:54.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:54.713 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:54.711+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:54.815 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:54.813+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:54.830 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:29:54.830 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:29:54.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.538547994, 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:29:54.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:54.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:29:54.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.838 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.836629059, 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.839 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:54.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:29:54.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.83660655, 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.846 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:29:54.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.83656022300001, 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:29:54.853 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:29:54.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:29:54.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:29:54.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.836393466, 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:29:54.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:29:54.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.836378701, 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.868 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:29:54.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:29:54.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.491056384, 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.875 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:29:54.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:29:54.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:29:54.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:29:54.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.49101870600001, 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:29:54.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:29:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.490995893, 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:29:54.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:29:54.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:29:54.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:54.895 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.49091068, 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:29:54.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:29:54.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:29:54.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:29:54.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:29:54.902 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 80.184331755000002, 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:29:54.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 80.158423521000003, 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:29:54.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:29:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 80.089658365999995, 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:29:54.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:29:54.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.588115549000001, 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:29:54.918 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:29:54.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.393190229999998, 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.923 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:29:54.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:29:54.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:29:54.928 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 22.237313194999999, 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:54.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 20.958361365000002, 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.934 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:29:54.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:54.939 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.183006658, 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:29:54.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.085736237000001, 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:29:54.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:29:54.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.084054984, 2024-09-16T13:29:54.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:29:54.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:29:54.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:29:54.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:29:54.955 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:29:54.957 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:54.954+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:54.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:55.061 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.061 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:55.061 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:55.061 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.061 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:55.056+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:55.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:55.159 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:55.156+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:55.165 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.165 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:55.165 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:55.165 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:55.165 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:55.259 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:55.257+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.145292354999995, 2024-09-16T13:29:55.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:29:55.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.139816789999998, 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:29:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:29:55.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.379489982999999, 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.326 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:29:55.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:29:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.362951858000002, 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.332 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:29:55.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:29:55.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.020248082999998, 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:29:55.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.020232278999998, 2024-09-16T13:29:55.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:29:55.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:29:55.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.020216718, 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.346 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:29:55.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:29:55.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.020199004000006, 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:29:55.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.020172420999998, 2024-09-16T13:29:55.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:29:55.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 76.178618508, 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:29:55.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:29:55.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.981178936999996, 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:29:55.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:29:55.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 55.406302330999999, 2024-09-16T13:29:55.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:29:55.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.376 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.059012373000002, 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:29:55.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.058732477, 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:29:55.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:29:55.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.627040761000003, 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:29:55.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:29:55.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.032969379000001, 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:29:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:29:55.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.964239677999998, 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:29:55.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:29:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:29:55.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:29:55.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.953374939, 2024-09-16T13:29:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:29:55.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100190 4.1e 4:7a9178f5:::rbd_data.11a97e9c0852.0000000000000012:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:29:55.412 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.509291146000001, 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048355217999999998, 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100190 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:29:55.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801667+0000", 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801670+0000", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801678+0000", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3119999999999996e-06 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801681+0000", 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2024-09-16T13:29:55.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801710+0000", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9496999999999999e-05 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801752+0000", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1288000000000002e-05 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.804108+0000", 2024-09-16T13:29:55.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023560870000000002 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849951+0000", 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045842898999999999 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849959+0000", 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5180000000000004e-06 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849995+0000", 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6241000000000001e-05 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.850022+0000", 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6336999999999999e-05 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.418 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103827 4.1e 4:7b9e4072:::rbd_data.11a976e61a66.0000000000000020:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.1182457140000004, 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046988757999999999, 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103827 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192713+0000", 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9000000000000005e-07 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192718+0000", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7060000000000003e-06 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192728+0000", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0730999999999999e-05 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192733+0000", 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6079999999999998e-06 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192774+0000", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0859999999999998e-05 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192861+0000", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7551000000000003e-05 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.195707+0000", 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028459779999999999 2024-09-16T13:29:55.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239591+0000", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043883414000000003 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239603+0000", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1843e-05 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239660+0000", 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7012000000000002e-05 2024-09-16T13:29:55.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239701+0000", 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1065000000000001e-05 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.424 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:55.426 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:55.424+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:55.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:29:55.526 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:55.525+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:55.585 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.585 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:55.585 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:55.585 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:55.627 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:55.626+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:55.693 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:55.693 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.701 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:55.728 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:55.727+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:55.802 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.097232840999993, 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.076168933000005, 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.169198672999997, 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013265568000001, 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013249698999999, 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013234662000002, 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013219995, 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013205253999999, 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013187434999999, 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.013003028, 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.012984932000002, 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.012964085999997, 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.012949290999998, 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.012934684000001, 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.012920227999999, 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.517029791999999, 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:29:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.067759782, 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:29:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.4256179749999998, 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:29:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:29:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:29:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:29:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.0511873420000004, 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:29:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:29:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:29:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:29:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.0397854520000003, 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:29:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:55.900 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:55.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:55.902 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:55.899+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:56.000 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:56.001 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:56.002 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:56.001+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:56.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:56.103 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:56.100+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:56.106 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:56.107 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:56.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:56.203 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:56.201+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:56.255 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:56.255 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:56.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:56.255 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:56.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.120822101000002, 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.256 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:56.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:56.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.730004186999999, 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:56.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:56.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.092329718000002, 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.937297680999997, 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:56.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.932002404000002, 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:56.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:56.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.931443846000001, 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.286 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:56.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.258790343000001, 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.292 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:56.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.258764098, 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.297 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:56.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:56.301 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.258744190000002, 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:56.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.682266187000003, 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.306 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:56.311 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.516918549000003, 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.312 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:56.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.500725803999998, 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.318 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:56.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:56.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:56.322 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.374007902999999, 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:56.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:29:56.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:29:56.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:29:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.300177764000001, 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:29:56.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.181436527999999, 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.333 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:29:56.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.169264839, 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:29:56.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.61450634, 2024-09-16T13:29:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:29:56.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:29:56.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7697276809999991, 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.347 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:29:56.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.7599312719999993, 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.353 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:29:56.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:29:56.357 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 5.817031064, 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:56.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:29:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:56.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:29:56.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:56.364 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:56.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:56.366 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:56.363+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:56.466 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:56.464+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:56.478 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:29:56.512 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:56.513 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:56.513 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:56.513 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:56.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:29:56.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:29:56.606 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:56.603+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:56.664 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:56.664 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:56.664 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:56.664 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:56.664 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:56.668 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:56.666+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:56.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:29:56.721 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:56.737 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:56.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:56.752 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:56.767 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:56.769 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:56.767+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:56.783 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:56.783 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:56.799 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:56.800 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:56.813 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:56.813 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:56.815 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:56.815 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:56.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:29:56.831 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:56.831 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:56.870 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:56.869+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:56.931 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:56.932 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:56.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:29:56.972 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:56.970+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:57.072 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:57.071+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:57.079 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.079 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.087 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:29:57.173 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:57.172+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:57.226 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.226 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:29:57.276 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:57.274+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:57.373 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.373 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.375 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:57.374+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:57.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:29:57.477 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:57.475+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:57.528 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.528 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:29:57.579 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:57.577+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:57.676 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.676 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.678 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:57.676+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:57.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:29:57.779 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:57.778+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:57.823 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.823 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:29:57.916 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:57.914+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:57.969 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:57.970 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:57.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:29:58.019 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:58.017+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:58.082 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:58.081+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:58.116 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.117 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:29:58.184 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:58.182+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:58.262 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.262 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:29:58.285 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:58.283+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:58.368 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.368 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.376 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:29:58.420 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:29:58.418+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:29:58.473 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.474 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:29:58.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:29:58.527 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:58.525+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:58.586 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:58.584+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:58.607 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.608 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:29:58.703 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:58.701+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:58.757 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.758 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:29:58.788 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:29:58.787+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:29:58.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:29:58.893 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:58.892+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:58.907 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:29:58.907 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:29:58.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:29:58.965 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:29:58.966 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:29:58.982 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:29:58.982 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:29:58.989 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:58.988+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:58.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:29:58.997 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:29:59.025 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:29:59.026 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:29:59.046 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:29:59.046 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:29:59.061 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:29:59.061 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:29:59.070 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:29:59.071 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.076 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:29:59.076 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:29:59.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:29:59.091 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:59.089+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:59.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:29:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:29:59.190 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:59.190 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.192 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:29:59.190+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:29:59.198 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:29:59.293 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:29:59.292+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:29:59.339 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 80.633967681000001, 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:29:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:29:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 80.612903772999999, 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:29:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:29:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.705933513000005, 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:29:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.550000408000002, 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:29:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549984539, 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:29:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549969502000003, 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:29:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:29:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549954835000001, 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:29:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:29:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549940094, 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:29:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:29:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549922275, 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:29:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:29:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:29:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549737868000001, 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:29:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:29:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549719772000003, 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:29:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:29:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549698925999998, 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549684130999999, 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:29:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:29:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:29:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549669524000002, 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.549655068, 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:29:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:29:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.053764631999996, 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:29:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:29:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.604494622000001, 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:29:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:29:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.962352814999999, 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:29:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:29:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.587922182, 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:29:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.576520292, 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.440 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:29:59.442 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:59.439+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:59.480 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:29:59.481 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:29:59.481 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T13:29:59.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:29:59.490 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:29:59.548 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:29:59.547+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:29:59.601 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:107942 4.1c 4:380f7ae3:::rbd_header.11a98de10525:head [call rbd.set_protection_status in=33b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:59.598356+0000", 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.0015128030000000001, 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015471879999999999, 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 107942 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598356+0000", 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598356+0000", 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598356+0000", 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5199999999999995e-07 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598361+0000", 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9479999999999999e-06 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598375+0000", 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3808000000000001e-05 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598382+0000", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2920000000000001e-06 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598421+0000", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9734999999999998e-05 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598498+0000", 2024-09-16T13:29:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6402000000000005e-05 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:59.598710+0000", 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021280299999999999 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 1 2024-09-16T13:29:59.607 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:29:59.643 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:29:59.640+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:29:59.744 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:59.742+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:29:59.752 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:29:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:29:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:29:59.752 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:29:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:29:59.772 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 111, 2024-09-16T13:29:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:29:59.482782+0000", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:29:06.165279+0000", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:29:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 4, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:29:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:29:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:29:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:29:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:29:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:29:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:29:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "pool_client.0", 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:25.583589+0000", 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:29:59.792 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:29:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "111", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 57, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 111, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:29:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:29:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:29:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:29:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.803 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:29:59.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:29:59.808 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.6766510009765625, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.814 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:29:59.815 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:29:59.820 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.821 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.826 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 96, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.827 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:29:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:29:59.833 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:29:59.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:29:59.838 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:29:59.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:29:59.843 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.844 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.846 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.847 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:29:59.849 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.5", 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.d", 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.2", 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.851 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.853 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 38, 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 10 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2024-09-16T13:29:59.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 38, 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:29:59.855 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.856 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.858 INFO:tasks.thrashosds.thrasher:j is {'epoch': 111, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:29:59.482782+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:29:06.165279+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:28:25.583589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 57, 'snap_epoch': 111, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6766510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [6, 1]}, {'pgid': '4.d', 'osds': [5, 0]}], 'pg_upmap_items': [{'pgid': '2.2', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 38, 'length': 10}]}], 'new_removed_snaps': [{'pool': 4, 'snaps': [{'begin': 38, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:29:59.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:29:59.858 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:29:59.855+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:29:59.860 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 85.726417472999998, 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:29:59.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 55.335599559000002, 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:29:59.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:29:59.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:29:59.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.697925089999998, 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.873 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:29:59.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:29:59.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.542893053, 2024-09-16T13:29:59.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:29:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:29:59.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.537597775999998, 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.884 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:29:59.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:29:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.537039217999997, 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:29:59.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.864385714999997, 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:29:59.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:29:59.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.864359469999997, 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.904 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.864339561999998, 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:29:59.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:59.909 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.287861559, 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:59.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.122513920999999, 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:29:59.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.106321176000002, 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.921 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:29:59.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.979603274999999, 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:29:59.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:29:59.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:29:59.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.905773136000001, 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.931 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:29:59.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.787031899999999, 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:29:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:29:59.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:29:59.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 25.774860211, 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:29:59.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:29:59.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.220101712000002, 2024-09-16T13:29:59.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:29:59.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:29:59.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.375323053000001, 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.949 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:29:59.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:29:59.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:29:59.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:29:59.954 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.365526643999999, 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:29:59.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:59.958 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.4226264359999998, 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:59.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:29:59.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:29:59.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:29:59.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:29:59.965 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:29:59.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:29:59.967 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:29:59.963+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:00.067 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.065+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:00.119 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:00.119 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:00.119 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:00.119 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:00.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:00.169 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:00.166+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:00.263 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:00.264 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:00.269 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:00.267+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:00.271 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:00.271 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:00.271 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:00.271 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:00.271 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:00.275 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T13:29:59.922636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1378299455,"num_objects":1865,"num_object_clones":488,"num_object_copies":3730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":711,"num_read":153348,"num_read_kb":793370,"num_write":19902,"num_write_kb":462097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":38,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15529,"ondisk_log_size":15529,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2503128,"kb_used_data":70176,"kb_used_omap":13,"kb_used_meta":2432818,"kb_avail":752471592,"statfs":{"total":773094113280,"available":770530910208,"internally_reserved":0,"allocated":71860224,"data_stored":590334920,"data_compressed":14835834,"data_compressed_allocated":48984064,"data_compressed_original":577100665,"omap_allocated":13632,"internal_metadata":2491206336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-216198630,"num_objects":-241,"num_object_clones":-72,"num_object_copies":-482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-242,"num_whiteouts":-72,"num_read":1208,"num_read_kb":1041,"num_write":266,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.449202"},"pg_stats":[{"pgid":"4.1f","version":"109'309","reported_seq":1892,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.554479+0000","last_change":"2024-09-16T13:29:57.554479+0000","last_active":"2024-09-16T13:29:57.554479+0000","last_peered":"2024-09-16T13:29:57.554479+0000","last_clean":"2024-09-16T13:29:57.554479+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:29:57.554479+0000","last_undegraded":"2024-09-16T13:29:57.554479+0000","last_fullsized":"2024-09-16T13:29:57.554479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030870700000000002,"stat_sum":{"num_bytes":51906101,"num_objects":46,"num_object_clones":11,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":20,"num_read":2544,"num_read_kb":19874,"num_write":326,"num_write_kb":9426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.1e","version":"109'981","reported_seq":11393,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.553990+0000","last_change":"2024-09-16T13:29:57.479889+0000","last_active":"2024-09-16T13:29:57.553990+0000","last_peered":"2024-09-16T13:29:57.553990+0000","last_clean":"2024-09-16T13:29:57.553990+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:29:57.553990+0000","last_undegraded":"2024-09-16T13:29:57.553990+0000","last_fullsized":"2024-09-16T13:29:57.553990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000570029,"stat_sum":{"num_bytes":58800777,"num_objects":56,"num_object_clones":13,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":21,"num_read":14604,"num_read_kb":30757,"num_write":1580,"num_write_kb":9668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.1d","version":"110'379","reported_seq":1307,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.601153+0000","last_change":"2024-09-16T13:29:58.493582+0000","last_active":"2024-09-16T13:29:58.601153+0000","last_peered":"2024-09-16T13:29:58.601153+0000","last_clean":"2024-09-16T13:29:58.601153+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:29:58.601153+0000","last_undegraded":"2024-09-16T13:29:58.601153+0000","last_fullsized":"2024-09-16T13:29:58.601153+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027583130000000001,"stat_sum":{"num_bytes":35952187,"num_objects":47,"num_object_clones":13,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":23,"num_read":1772,"num_read_kb":7929,"num_write":409,"num_write_kb":5823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.1c","version":"109'367","reported_seq":3564,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.535353+0000","last_change":"2024-09-16T13:29:57.491839+0000","last_active":"2024-09-16T13:29:57.535353+0000","last_peered":"2024-09-16T13:29:57.535353+0000","last_clean":"2024-09-16T13:29:57.535353+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:29:57.535353+0000","last_undegraded":"2024-09-16T13:29:57.535353+0000","last_fullsized":"2024-09-16T13:29:57.535353+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048322600000000001,"stat_sum":{"num_bytes":29769192,"num_objects":47,"num_object_clones":12,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":14,"num_read":6038,"num_read_kb":25523,"num_write":376,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.1b","version":"108'418","reported_seq":2113,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.485094+0000","last_change":"2024-09-16T13:29:58.485280+0000","last_active":"2024-09-16T13:29:58.485094+0000","last_peered":"2024-09-16T13:29:58.485094+0000","last_clean":"2024-09-16T13:29:58.485094+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:29:58.485094+0000","last_undegraded":"2024-09-16T13:29:58.485094+0000","last_fullsized":"2024-09-16T13:29:58.485094+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064065499999999996,"stat_sum":{"num_bytes":32346319,"num_objects":53,"num_object_clones":15,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":20,"num_read":3206,"num_read_kb":15604,"num_write":505,"num_write_kb":9032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.1a","version":"109'357","reported_seq":3717,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.617270+0000","last_change":"2024-09-16T13:29:58.484922+0000","last_active":"2024-09-16T13:29:58.617270+0000","last_peered":"2024-09-16T13:29:58.617270+0000","last_clean":"2024-09-16T13:29:58.617270+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:29:58.617270+0000","last_undegraded":"2024-09-16T13:29:58.617270+0000","last_fullsized":"2024-09-16T13:29:58.617270+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039033299999999998,"stat_sum":{"num_bytes":36853618,"num_objects":51,"num_object_clones":10,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":17,"num_read":3720,"num_read_kb":16180,"num_write":412,"num_write_kb":6959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.19","version":"110'320","reported_seq":3169,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.628909+0000","last_change":"2024-09-16T13:29:58.492692+0000","last_active":"2024-09-16T13:29:58.628909+0000","last_peered":"2024-09-16T13:29:58.628909+0000","last_clean":"2024-09-16T13:29:58.628909+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:29:58.628909+0000","last_undegraded":"2024-09-16T13:29:58.628909+0000","last_fullsized":"2024-09-16T13:29:58.628909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022007379999999998,"stat_sum":{"num_bytes":54990339,"num_objects":78,"num_object_clones":22,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":29,"num_read":3291,"num_read_kb":15747,"num_write":360,"num_write_kb":23287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.18","version":"110'335","reported_seq":3194,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.485524+0000","last_change":"2024-09-16T13:29:58.485524+0000","last_active":"2024-09-16T13:29:58.485524+0000","last_peered":"2024-09-16T13:29:58.485524+0000","last_clean":"2024-09-16T13:29:58.485524+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:29:58.485524+0000","last_undegraded":"2024-09-16T13:29:58.485524+0000","last_fullsized":"2024-09-16T13:29:58.485524+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010110379999999999,"stat_sum":{"num_bytes":58653288,"num_objects":60,"num_object_clones":18,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":18,"num_read":3436,"num_read_kb":32798,"num_write":371,"num_write_kb":30744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.17","version":"109'319","reported_seq":2727,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.492202+0000","last_change":"2024-09-16T13:29:58.492432+0000","last_active":"2024-09-16T13:29:58.492202+0000","last_peered":"2024-09-16T13:29:58.492202+0000","last_clean":"2024-09-16T13:29:58.492202+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:29:58.492202+0000","last_undegraded":"2024-09-16T13:29:58.492202+0000","last_fullsized":"2024-09-16T13:29:58.492202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020287259999999998,"stat_sum":{"num_bytes":35593455,"num_objects":64,"num_object_clones":18,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":31,"num_read":3769,"num_read_kb":29185,"num_write":318,"num_write_kb":17559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.16","version":"109'558","reported_seq":2383,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.558644+0000","last_change":"2024-09-16T13:29:57.480300+0000","last_active":"2024-09-16T13:29:57.558644+0000","last_peered":"2024-09-16T13:29:57.558644+0000","last_clean":"2024-09-16T13:29:57.558644+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:29:57.558644+0000","last_undegraded":"2024-09-16T13:29:57.558644+0000","last_fullsized":"2024-09-16T13:29:57.558644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078074000000000004,"stat_sum":{"num_bytes":16848701,"num_objects":40,"num_object_clones":7,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":13,"num_read":4110,"num_read_kb":15480,"num_write":719,"num_write_kb":9968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.15","version":"110'737","reported_seq":3758,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.588975+0000","last_change":"2024-09-16T13:29:58.588975+0000","last_active":"2024-09-16T13:29:58.588975+0000","last_peered":"2024-09-16T13:29:58.588975+0000","last_clean":"2024-09-16T13:29:58.588975+0000","last_became_active":"2024-09-16T13:29:55.466429+0000","last_became_peered":"2024-09-16T13:29:55.466429+0000","last_unstale":"2024-09-16T13:29:58.588975+0000","last_undegraded":"2024-09-16T13:29:58.588975+0000","last_fullsized":"2024-09-16T13:29:58.588975+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:31.786154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011799429999999999,"stat_sum":{"num_bytes":71841911,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":22,"num_read":5516,"num_read_kb":34124,"num_write":1163,"num_write_kb":18006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.14","version":"110'402","reported_seq":3068,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.629283+0000","last_change":"2024-09-16T13:29:58.493673+0000","last_active":"2024-09-16T13:29:58.629283+0000","last_peered":"2024-09-16T13:29:58.629283+0000","last_clean":"2024-09-16T13:29:58.629283+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:29:58.629283+0000","last_undegraded":"2024-09-16T13:29:58.629283+0000","last_fullsized":"2024-09-16T13:29:58.629283+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027248810000000002,"stat_sum":{"num_bytes":12421621,"num_objects":53,"num_object_clones":14,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":23,"num_read":4540,"num_read_kb":20151,"num_write":456,"num_write_kb":8915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.13","version":"108'321","reported_seq":1765,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.485190+0000","last_change":"2024-09-16T13:29:58.485360+0000","last_active":"2024-09-16T13:29:58.485190+0000","last_peered":"2024-09-16T13:29:58.485190+0000","last_clean":"2024-09-16T13:29:58.485190+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:29:58.485190+0000","last_undegraded":"2024-09-16T13:29:58.485190+0000","last_fullsized":"2024-09-16T13:29:58.485190+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062651199999999997,"stat_sum":{"num_bytes":45757587,"num_objects":65,"num_object_clones":19,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":28,"num_read":1514,"num_read_kb":22417,"num_write":332,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.12","version":"109'687","reported_seq":3535,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.606227+0000","last_change":"2024-09-16T13:29:57.482720+0000","last_active":"2024-09-16T13:29:57.606227+0000","last_peered":"2024-09-16T13:29:57.606227+0000","last_clean":"2024-09-16T13:29:57.606227+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:29:57.606227+0000","last_undegraded":"2024-09-16T13:29:57.606227+0000","last_fullsized":"2024-09-16T13:29:57.606227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044695500000000002,"stat_sum":{"num_bytes":14427395,"num_objects":48,"num_object_clones":10,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":20,"num_read":7564,"num_read_kb":19653,"num_write":890,"num_write_kb":8801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.11","version":"109'700","reported_seq":4578,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.490818+0000","last_change":"2024-09-16T13:29:58.491141+0000","last_active":"2024-09-16T13:29:58.490818+0000","last_peered":"2024-09-16T13:29:58.490818+0000","last_clean":"2024-09-16T13:29:58.490818+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:29:58.490818+0000","last_undegraded":"2024-09-16T13:29:58.490818+0000","last_fullsized":"2024-09-16T13:29:58.490818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083534199999999999,"stat_sum":{"num_bytes":47185183,"num_objects":71,"num_object_clones":24,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":30,"num_read":5901,"num_read_kb":16214,"num_write":930,"num_write_kb":25099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.10","version":"108'629","reported_seq":3166,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.490742+0000","last_change":"2024-09-16T13:29:58.491398+0000","last_active":"2024-09-16T13:29:58.490742+0000","last_peered":"2024-09-16T13:29:58.490742+0000","last_clean":"2024-09-16T13:29:58.490742+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:29:58.490742+0000","last_undegraded":"2024-09-16T13:29:58.490742+0000","last_fullsized":"2024-09-16T13:29:58.490742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011693529999999999,"stat_sum":{"num_bytes":28268662,"num_objects":45,"num_object_clones":9,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":19,"num_read":7126,"num_read_kb":20209,"num_write":903,"num_write_kb":9509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"2.7","version":"0'0","reported_seq":119,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.554091+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:29:56.554091+0000","last_peered":"2024-09-16T13:29:56.554091+0000","last_clean":"2024-09-16T13:29:56.554091+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:29:56.554091+0000","last_undegraded":"2024-09-16T13:29:56.554091+0000","last_fullsized":"2024-09-16T13:29:56.554091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"110'424","reported_seq":3335,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.485639+0000","last_change":"2024-09-16T13:29:58.485639+0000","last_active":"2024-09-16T13:29:58.485639+0000","last_peered":"2024-09-16T13:29:58.485639+0000","last_clean":"2024-09-16T13:29:58.485639+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:29:58.485639+0000","last_undegraded":"2024-09-16T13:29:58.485639+0000","last_fullsized":"2024-09-16T13:29:58.485639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001120998,"stat_sum":{"num_bytes":37754876,"num_objects":74,"num_object_clones":25,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":33,"num_read":3466,"num_read_kb":37800,"num_write":564,"num_write_kb":17612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.476658+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:56.476658+0000","last_peered":"2024-09-16T13:29:56.476658+0000","last_clean":"2024-09-16T13:29:56.476658+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:29:56.476658+0000","last_undegraded":"2024-09-16T13:29:56.476658+0000","last_fullsized":"2024-09-16T13:29:56.476658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"108'291","reported_seq":2373,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.492132+0000","last_change":"2024-09-16T13:29:58.492398+0000","last_active":"2024-09-16T13:29:58.492132+0000","last_peered":"2024-09-16T13:29:58.492132+0000","last_clean":"2024-09-16T13:29:58.492132+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:29:58.492132+0000","last_undegraded":"2024-09-16T13:29:58.492132+0000","last_fullsized":"2024-09-16T13:29:58.492132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019978589999999998,"stat_sum":{"num_bytes":32268823,"num_objects":66,"num_object_clones":16,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":24,"num_read":2274,"num_read_kb":15213,"num_write":391,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.553898+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:29:56.553898+0000","last_peered":"2024-09-16T13:29:56.553898+0000","last_clean":"2024-09-16T13:29:56.553898+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:29:56.553898+0000","last_undegraded":"2024-09-16T13:29:56.553898+0000","last_fullsized":"2024-09-16T13:29:56.553898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"109'600","reported_seq":3674,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.582671+0000","last_change":"2024-09-16T13:29:57.480072+0000","last_active":"2024-09-16T13:29:57.582671+0000","last_peered":"2024-09-16T13:29:57.582671+0000","last_clean":"2024-09-16T13:29:57.582671+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:29:57.582671+0000","last_undegraded":"2024-09-16T13:29:57.582671+0000","last_fullsized":"2024-09-16T13:29:57.582671+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061793000000000004,"stat_sum":{"num_bytes":58398808,"num_objects":72,"num_object_clones":21,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":25,"num_read":4781,"num_read_kb":49383,"num_write":719,"num_write_kb":31463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.476578+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:29:56.476578+0000","last_peered":"2024-09-16T13:29:56.476578+0000","last_clean":"2024-09-16T13:29:56.476578+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:29:56.476578+0000","last_undegraded":"2024-09-16T13:29:56.476578+0000","last_fullsized":"2024-09-16T13:29:56.476578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"109'528","reported_seq":3048,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.575145+0000","last_change":"2024-09-16T13:29:57.483560+0000","last_active":"2024-09-16T13:29:57.575145+0000","last_peered":"2024-09-16T13:29:57.575145+0000","last_clean":"2024-09-16T13:29:57.575145+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:29:57.575145+0000","last_undegraded":"2024-09-16T13:29:57.575145+0000","last_fullsized":"2024-09-16T13:29:57.575145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113579,"stat_sum":{"num_bytes":17592455,"num_objects":45,"num_object_clones":10,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":11,"num_read":5816,"num_read_kb":17907,"num_write":617,"num_write_kb":26206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"3.3","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.495919+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:29:57.495919+0000","last_peered":"2024-09-16T13:29:57.495919+0000","last_clean":"2024-09-16T13:29:57.495919+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:29:57.495919+0000","last_undegraded":"2024-09-16T13:29:57.495919+0000","last_fullsized":"2024-09-16T13:29:57.495919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":27,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.476022+0000","last_change":"2024-09-16T13:29:55.469146+0000","last_active":"2024-09-16T13:29:56.476022+0000","last_peered":"2024-09-16T13:29:56.476022+0000","last_clean":"2024-09-16T13:29:56.476022+0000","last_became_active":"2024-09-16T13:29:55.468831+0000","last_became_peered":"2024-09-16T13:29:55.468831+0000","last_unstale":"2024-09-16T13:29:56.476022+0000","last_undegraded":"2024-09-16T13:29:56.476022+0000","last_fullsized":"2024-09-16T13:29:56.476022+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:56.220967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"109'316","reported_seq":4299,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.482818+0000","last_change":"2024-09-16T13:29:57.482818+0000","last_active":"2024-09-16T13:29:57.482818+0000","last_peered":"2024-09-16T13:29:57.482818+0000","last_clean":"2024-09-16T13:29:57.482818+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:29:57.482818+0000","last_undegraded":"2024-09-16T13:29:57.482818+0000","last_fullsized":"2024-09-16T13:29:57.482818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047949700000000001,"stat_sum":{"num_bytes":34126827,"num_objects":44,"num_object_clones":13,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":15,"num_read":4210,"num_read_kb":35919,"num_write":364,"num_write_kb":9719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"3.0","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.475225+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:29:56.475225+0000","last_peered":"2024-09-16T13:29:56.475225+0000","last_clean":"2024-09-16T13:29:56.475225+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:29:56.475225+0000","last_undegraded":"2024-09-16T13:29:56.475225+0000","last_fullsized":"2024-09-16T13:29:56.475225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.493133+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:29:56.493133+0000","last_peered":"2024-09-16T13:29:56.493133+0000","last_clean":"2024-09-16T13:29:56.493133+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:29:56.493133+0000","last_undegraded":"2024-09-16T13:29:56.493133+0000","last_fullsized":"2024-09-16T13:29:56.493133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"109'388","reported_seq":2990,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.482534+0000","last_change":"2024-09-16T13:29:57.482534+0000","last_active":"2024-09-16T13:29:57.482534+0000","last_peered":"2024-09-16T13:29:57.482534+0000","last_clean":"2024-09-16T13:29:57.482534+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:29:57.482534+0000","last_undegraded":"2024-09-16T13:29:57.482534+0000","last_fullsized":"2024-09-16T13:29:57.482534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027629800000000002,"stat_sum":{"num_bytes":59265421,"num_objects":69,"num_object_clones":20,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":24,"num_read":4081,"num_read_kb":20017,"num_write":456,"num_write_kb":8571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"3.1","version":"105'2","reported_seq":94,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.474857+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:29:56.474857+0000","last_peered":"2024-09-16T13:29:56.474857+0000","last_clean":"2024-09-16T13:29:56.474857+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:29:56.474857+0000","last_undegraded":"2024-09-16T13:29:56.474857+0000","last_fullsized":"2024-09-16T13:29:56.474857+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.478004+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:29:57.478004+0000","last_peered":"2024-09-16T13:29:57.478004+0000","last_clean":"2024-09-16T13:29:57.478004+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:29:57.478004+0000","last_undegraded":"2024-09-16T13:29:57.478004+0000","last_fullsized":"2024-09-16T13:29:57.478004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"109'452","reported_seq":3523,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.480963+0000","last_change":"2024-09-16T13:29:57.480963+0000","last_active":"2024-09-16T13:29:57.480963+0000","last_peered":"2024-09-16T13:29:57.480963+0000","last_clean":"2024-09-16T13:29:57.480963+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:29:57.480963+0000","last_undegraded":"2024-09-16T13:29:57.480963+0000","last_fullsized":"2024-09-16T13:29:57.480963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001311666,"stat_sum":{"num_bytes":94433618,"num_objects":67,"num_object_clones":17,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":4990,"num_read_kb":40576,"num_write":483,"num_write_kb":30169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"3.2","version":"101'3","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.479024+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:29:57.479024+0000","last_peered":"2024-09-16T13:29:57.479024+0000","last_clean":"2024-09-16T13:29:57.479024+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:29:57.479024+0000","last_undegraded":"2024-09-16T13:29:57.479024+0000","last_fullsized":"2024-09-16T13:29:57.479024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":85,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T13:29:56.493010+0000","last_change":"2024-09-16T13:29:55.466745+0000","last_active":"2024-09-16T13:29:56.493010+0000","last_peered":"2024-09-16T13:29:56.493010+0000","last_clean":"2024-09-16T13:29:56.493010+0000","last_became_active":"2024-09-16T13:29:55.466418+0000","last_became_peered":"2024-09-16T13:29:55.466418+0000","last_unstale":"2024-09-16T13:29:56.493010+0000","last_undegraded":"2024-09-16T13:29:56.493010+0000","last_fullsized":"2024-09-16T13:29:56.493010+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:33.757459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.477938+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:29:57.477938+0000","last_peered":"2024-09-16T13:29:57.477938+0000","last_clean":"2024-09-16T13:29:57.477938+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:29:57.477938+0000","last_undegraded":"2024-09-16T13:29:57.477938+0000","last_fullsized":"2024-09-16T13:29:57.477938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"109'518","reported_seq":2138,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.554428+0000","last_change":"2024-09-16T13:29:57.554428+0000","last_active":"2024-09-16T13:29:57.554428+0000","last_peered":"2024-09-16T13:29:57.554428+0000","last_clean":"2024-09-16T13:29:57.554428+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:29:57.554428+0000","last_undegraded":"2024-09-16T13:29:57.554428+0000","last_fullsized":"2024-09-16T13:29:57.554428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026026499999999998,"stat_sum":{"num_bytes":30560340,"num_objects":57,"num_object_clones":13,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":20,"num_read":3235,"num_read_kb":17040,"num_write":676,"num_write_kb":7420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.8","version":"109'1152","reported_seq":5296,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.587907+0000","last_change":"2024-09-16T13:29:58.588209+0000","last_active":"2024-09-16T13:29:58.587907+0000","last_peered":"2024-09-16T13:29:58.587907+0000","last_clean":"2024-09-16T13:29:58.587907+0000","last_became_active":"2024-09-16T13:29:55.466349+0000","last_became_peered":"2024-09-16T13:29:55.466349+0000","last_unstale":"2024-09-16T13:29:58.587907+0000","last_undegraded":"2024-09-16T13:29:58.587907+0000","last_fullsized":"2024-09-16T13:29:58.587907+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:08.619344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043241999999999998,"stat_sum":{"num_bytes":39812956,"num_objects":56,"num_object_clones":13,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":15,"num_read":9206,"num_read_kb":25287,"num_write":1751,"num_write_kb":11378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.9","version":"110'304","reported_seq":4367,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.484850+0000","last_change":"2024-09-16T13:29:58.484850+0000","last_active":"2024-09-16T13:29:58.484850+0000","last_peered":"2024-09-16T13:29:58.484850+0000","last_clean":"2024-09-16T13:29:58.484850+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:29:58.484850+0000","last_undegraded":"2024-09-16T13:29:58.484850+0000","last_fullsized":"2024-09-16T13:29:58.484850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000336297,"stat_sum":{"num_bytes":72416069,"num_objects":60,"num_object_clones":20,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":18,"num_read":5372,"num_read_kb":29660,"num_write":392,"num_write_kb":9459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.a","version":"107'432","reported_seq":4040,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.554454+0000","last_change":"2024-09-16T13:29:57.554574+0000","last_active":"2024-09-16T13:29:57.554454+0000","last_peered":"2024-09-16T13:29:57.554454+0000","last_clean":"2024-09-16T13:29:57.554454+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:29:57.554454+0000","last_undegraded":"2024-09-16T13:29:57.554454+0000","last_fullsized":"2024-09-16T13:29:57.554454+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000222267,"stat_sum":{"num_bytes":47448741,"num_objects":56,"num_object_clones":14,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":21,"num_read":5395,"num_read_kb":43127,"num_write":486,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.b","version":"109'354","reported_seq":5127,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.556570+0000","last_change":"2024-09-16T13:29:57.479805+0000","last_active":"2024-09-16T13:29:57.556570+0000","last_peered":"2024-09-16T13:29:57.556570+0000","last_clean":"2024-09-16T13:29:57.556570+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:29:57.556570+0000","last_undegraded":"2024-09-16T13:29:57.556570+0000","last_fullsized":"2024-09-16T13:29:57.556570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052301500000000005,"stat_sum":{"num_bytes":87349127,"num_objects":56,"num_object_clones":14,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":20,"num_read":5611,"num_read_kb":40287,"num_write":456,"num_write_kb":26179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.c","version":"109'407","reported_seq":5092,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.484901+0000","last_change":"2024-09-16T13:29:58.485051+0000","last_active":"2024-09-16T13:29:58.484901+0000","last_peered":"2024-09-16T13:29:58.484901+0000","last_clean":"2024-09-16T13:29:58.484901+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:29:58.484901+0000","last_undegraded":"2024-09-16T13:29:58.484901+0000","last_fullsized":"2024-09-16T13:29:58.484901+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050311300000000002,"stat_sum":{"num_bytes":48910395,"num_objects":63,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":25,"num_read":6213,"num_read_kb":25005,"num_write":506,"num_write_kb":8615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.d","version":"109'374","reported_seq":2882,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.585387+0000","last_change":"2024-09-16T13:29:57.480314+0000","last_active":"2024-09-16T13:29:57.585387+0000","last_peered":"2024-09-16T13:29:57.585387+0000","last_clean":"2024-09-16T13:29:57.585387+0000","last_became_active":"2024-09-16T13:29:55.469723+0000","last_became_peered":"2024-09-16T13:29:55.469723+0000","last_unstale":"2024-09-16T13:29:57.585387+0000","last_undegraded":"2024-09-16T13:29:57.585387+0000","last_fullsized":"2024-09-16T13:29:57.585387+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.171374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072002800000000005,"stat_sum":{"num_bytes":34895191,"num_objects":70,"num_object_clones":19,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":28,"num_read":2888,"num_read_kb":22083,"num_write":415,"num_write_kb":9806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"28","length":"8"}]},{"pgid":"4.e","version":"109'314","reported_seq":1535,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T13:29:58.484965+0000","last_change":"2024-09-16T13:29:58.485149+0000","last_active":"2024-09-16T13:29:58.484965+0000","last_peered":"2024-09-16T13:29:58.484965+0000","last_clean":"2024-09-16T13:29:58.484965+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:29:58.484965+0000","last_undegraded":"2024-09-16T13:29:58.484965+0000","last_fullsized":"2024-09-16T13:29:58.484965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056745400000000001,"stat_sum":{"num_bytes":22558182,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":27,"num_read":1670,"num_read_kb":17507,"num_write":388,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"9"}]},{"pgid":"4.f","version":"109'776","reported_seq":3150,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-16T13:29:57.491596+0000","last_change":"2024-09-16T13:29:57.491596+0000","last_active":"2024-09-16T13:29:57.491596+0000","last_peered":"2024-09-16T13:29:57.491596+0000","last_clean":"2024-09-16T13:29:57.491596+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:29:57.491596+0000","last_undegraded":"2024-09-16T13:29:57.491596+0000","last_fullsized":"2024-09-16T13:29:57.491596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028726899999999999,"stat_sum":{"num_bytes":28300735,"num_objects":59,"num_object_clones":17,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":26,"num_read":5423,"num_read_kb":14659,"num_write":1083,"num_write_kb":7898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"28","length":"8"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1377708900,"num_objects":1856,"num_object_clones":488,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":708,"num_read":153282,"num_read_kb":793315,"num_write":19797,"num_write_kb":460465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":60080128,"data_stored":568244732,"data_compressed":14496818,"data_compressed_allocated":47644672,"data_compressed_original":560184185,"omap_allocated":0,"internal_metadata":0},"log_size":15449,"ondisk_log_size":15449,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":168,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509467,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304272,"kb_used_data":6912,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067568,"statfs":{"total":96636764160,"available":96325189632,"internally_reserved":0,"allocated":7077888,"data_stored":49736321,"data_compressed":1268590,"data_compressed_allocated":4587520,"data_compressed_original":48344566,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.443},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.869,"5min":1.869,"15min":1.869},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.317}]},{"osd":5,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.435}]},{"osd":6,"last update":"Mon Sep 16 13:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.45400000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509469,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307144,"kb_used_data":9776,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94064696,"statfs":{"total":96636764160,"available":96322248704,"internally_reserved":0,"allocated":10010624,"data_stored":85387489,"data_compressed":2025023,"data_compressed_allocated":6877184,"data_compressed_original":83522184,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.169,"5min":2.169,"15min":2.169},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.371},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38100000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39200000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.25}]},{"osd":6,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.314},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.42299999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319452,"kb_used_data":9616,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94052388,"statfs":{"total":96636764160,"available":96309645312,"internally_reserved":0,"allocated":9846784,"data_stored":72301049,"data_compressed":2491368,"data_compressed_allocated":6811648,"data_compressed_original":70464759,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.442},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.34000000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.317}]},{"osd":6,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.371}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.5}]}]},{"osd":3,"up_from":15,"seq":64424509467,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310824,"kb_used_data":9312,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94061016,"statfs":{"total":96636764160,"available":96318480384,"internally_reserved":0,"allocated":9535488,"data_stored":61637648,"data_compressed":1979504,"data_compressed_allocated":6053888,"data_compressed_original":59634873,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.309},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.46400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.36899999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.49299999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317004,"kb_used_data":7168,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94054836,"statfs":{"total":96636764160,"available":96312152064,"internally_reserved":0,"allocated":7340032,"data_stored":67251407,"data_compressed":1467949,"data_compressed_allocated":4976640,"data_compressed_original":65949360,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.628},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.501}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.34899999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.39600000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309572,"kb_used_data":3892,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062268,"statfs":{"total":96636764160,"available":96319762432,"internally_reserved":0,"allocated":3985408,"data_stored":22634223,"data_compressed":736959,"data_compressed_allocated":2007040,"data_compressed_original":21549056,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.249},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":9.3390000000000004,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":9.4480000000000004,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.27100000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.4090000000000007,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:29:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":9.4299999999999997,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.74299999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.5640000000000001,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63300000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542173,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314996,"kb_used_data":13476,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94056844,"statfs":{"total":96636764160,"available":96314208256,"internally_reserved":0,"allocated":13799424,"data_stored":124813065,"data_compressed":3058034,"data_compressed_allocated":10088448,"data_compressed_original":122618353,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.379},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.54900000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542171,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319864,"kb_used_data":10024,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94051976,"statfs":{"total":96636764160,"available":96309223424,"internally_reserved":0,"allocated":10264576,"data_stored":106573718,"data_compressed":1808407,"data_compressed_allocated":7581696,"data_compressed_original":105017514,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26500000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.33800000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66400000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.441},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8269824,"data_stored":96202558,"data_compressed":1720310,"data_compressed_allocated":6914048,"data_compressed_original":95318186,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":124172947,"data_compressed":3058034,"data_compressed_allocated":10088448,"data_compressed_original":122618353,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2695168,"data_stored":21994124,"data_compressed":736959,"data_compressed_allocated":2007040,"data_compressed_original":21549056,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8237056,"data_stored":60997412,"data_compressed":1979504,"data_compressed_allocated":6053888,"data_compressed_original":59634873,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8380416,"data_stored":69694734,"data_compressed":2455627,"data_compressed_allocated":6656000,"data_compressed_original":68506871,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":81413881,"data_compressed":1886544,"data_compressed_allocated":6545408,"data_compressed_original":80196232,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6049792,"data_stored":66611308,"data_compressed":1467949,"data_compressed_allocated":4976640,"data_compressed_original":65949360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5591040,"data_stored":47157768,"data_compressed":1191891,"data_compressed_allocated":4403200,"data_compressed_original":46411254,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:00.278 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [4, 3, 0, 5] 2024-09-16T13:30:00.279 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.3', '4', '3', '0', '5'] 2024-09-16T13:30:00.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.3 4 3 0 5 2024-09-16T13:30:00.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:00.370 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.368+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:00.421 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:00.421 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:00.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:00.421 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:00.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175749836000001, 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175687363000002, 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.427 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:00.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:00.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:00.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175642709999998, 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.433 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:00.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:00.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:00.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175569609, 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:00.439 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:00.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:00.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:00.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175551102999997, 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.445 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:00.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:00.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:00.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:00.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175377150999999, 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.451 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:00.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:00.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.175338597, 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.457 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:00.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:00.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.462 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.174379903000002, 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:00.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.174322105999998, 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:00.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.174228229999997, 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:00.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.174200241000001, 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:00.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:00.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.172926214999997, 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:00.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:00.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:00.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:00.491 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.172902968999999, 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.44555682, 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.497 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.140928918, 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.128208088000001, 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:00.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:00.513 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.905174864000003, 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.425601786000001, 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:00.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:00.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.547885408999999, 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:00.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:00.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.530 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.530751109000001, 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:00.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:00.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:00.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.535 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:00.535 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:00.535 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:00.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:00.537 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.534+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:00.637 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:00.634+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:00.640 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:00.640 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:00.640 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:00.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:00.737 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.736+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:00.787 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:00.787 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:00.788 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:00.788 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:00.788 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:00.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:00.839 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.837+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:00.896 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:00.896 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 83.183316613000002, 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.897 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:30:00.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 83.173777501999993, 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:30:00.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 74.207320766999999, 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:30:00.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:30:00.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.496909316, 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.913 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:30:00.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.486262267000001, 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.819200686000002, 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:30:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:30:00.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.926 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.928 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.251569629999999, 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:30:00.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.934 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.234436735999999, 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:00.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.936 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.937 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.938 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.274047789999997, 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.941 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:00.942 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.944 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.257559069000003, 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.945 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.877674032000002, 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:30:00.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.877081102999998, 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.953 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:00.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.957 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.87706644, 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:30:00.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.582489788, 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:30:00.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.569208562, 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:30:00.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:30:00.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.603958886999999, 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.972 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:30:00.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:30:00.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.977 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.587682158, 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:30:00.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.981 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.316337244, 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.982 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:30:00.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.987 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.316142349, 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:30:00.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:30:00.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.296879431000001, 2024-09-16T13:30:00.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:30:00.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:00.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:00.997 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:00.999 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.3 pg_upmap_items mapping to [4->3,0->5] 2024-09-16T13:30:00.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:01.000 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:00.997+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:01.100 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:01.098+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:01.101 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:01.102 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:01.102 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:01.102 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:01.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:01.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:01.200 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:01.199+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:01.209 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:01.209 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:01.209 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:01.209 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:01.209 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:01.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:01.219 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:01.219 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:01.233 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:01.234 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:01.248 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:01.248 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:01.263 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:01.278 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:01.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:01.294 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:01.294 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:01.301 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:01.300+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:01.309 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:01.319 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.319 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.325 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:01.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:01.403 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:01.401+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:01.426 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.426 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:01.503 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:01.500+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:01.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:30:01.533 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.533 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:01.625 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:01.623+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:01.679 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.680 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:01.705 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:01.703+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:01.806 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:01.803+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:01.826 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.826 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:01.920 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:01.917+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:01.928 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:30:01.982 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:01.983 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:01.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:02.007 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:02.006+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:02.109 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:02.106+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:02.131 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.132 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:02.209 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:02.207+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:02.240 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.240 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:02.311 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:02.308+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:02.346 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.346 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:02.412 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:02.410+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:02.452 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.452 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:02.513 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:02.511+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:02.560 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.560 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:02.653 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:02.651+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:02.707 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.707 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:02.716 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:02.713+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:02.816 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:02.815+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:02.854 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:02.854 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:02.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:02.917 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:02.915+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:03.015 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:03.016 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:03.018 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:03.016+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:03.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:03.118 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:03.117+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:03.172 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:03.172 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:03.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:03.220 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:03.218+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:03.320 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:03.320 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:03.321 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:03.319+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:03.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:03.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:03.421 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:03.419+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:03.472 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:03.472 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:03.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:03.486 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:03.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:03.502 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:03.518 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:03.518 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:03.521 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:03.519+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:03.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:03.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:03.549 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:03.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:03.578 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:03.579 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:03.591 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:03.591 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:03.594 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:03.594 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:03.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:03.610 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:03.611 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:03.622 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:03.620+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464503632000003, 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464441159000003, 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464396506, 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464323405000002, 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464304898999998, 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464130947000001, 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.464092393000001, 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.463133698999997, 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.463075902, 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.462982025999999, 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.462954037000003, 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.461680010999999, 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.461656765000001, 2024-09-16T13:30:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.734310616000002, 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.429682714000002, 2024-09-16T13:30:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.416961884000003, 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.193928659999997, 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.714355582000003, 2024-09-16T13:30:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:03.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.836639205000001, 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.819504904999999, 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:03.825 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:03.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:03.826 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:03.824+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:03.927 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:03.924+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:03.938 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:03.938 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:03.938 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:03.939 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:03.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:30:03.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:04.036 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:04.033+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:04.094 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:04.094 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:04.094 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:04.094 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:04.094 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:04.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:04.144 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:04.142+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:04.229 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:04.227+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:04.244 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:04.244 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:04.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:04.244 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 86.530317565999994, 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:30:04.245 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.246 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.246 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:04.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:04.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:30:04.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:30:04.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:30:04.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:30:04.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:30:04.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:30:04.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:30:04.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:30:04.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:30:04.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:30:04.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:30:04.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.261 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 86.520778454999999, 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.262 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:04.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:30:04.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:04.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:30:04.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:30:04.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:30:04.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.554321720000004, 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.269 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:04.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:30:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:30:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.843910269000006, 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.276 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.277 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:30:04.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:30:04.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:30:04.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:30:04.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:30:04.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:30:04.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:30:04.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:30:04.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.833263220000006, 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.287 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:30:04.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:30:04.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 64.166201638999993, 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:30:04.291 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:30:04.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:30:04.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:30:04.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.598570582999997, 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.298 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:30:04.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:30:04.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.303 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.581437688999998, 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:30:04.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:30:04.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:30:04.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:30:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:30:04.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:30:04.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:04.312 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:04.312 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.621048743000003, 2024-09-16T13:30:04.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:30:04.312 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.312 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.313 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:30:04.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:30:04.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.320 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.604560022000001, 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:30:04.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.326 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.224674985, 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.327 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:30:04.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:30:04.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:30:04.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:30:04.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:30:04.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:30:04.336 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.336 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.336 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.337 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.337 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:04.337 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.224082056, 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.338 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:04.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:30:04.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:30:04.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:30:04.346 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.347 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.224067392999999, 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.348 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:30:04.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.354 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.929490740999999, 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:04.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:30:04.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:04.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.916209514999998, 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:30:04.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:04.364 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.950959839999999, 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:04.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:30:04.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:30:04.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.370 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.934683110999998, 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:30:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:30:04.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.663338197000002, 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.375 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.381 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.663143302000002, 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:30:04.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:30:04.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:30:04.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:04.388 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.643880383999999, 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:30:04.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:30:04.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.393 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:04.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:04.395 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:04.393+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:04.496 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:04.493+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:04.533 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:04.533 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:04.533 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:04.533 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:04.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:04.626 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:04.623+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:04.680 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:04.681 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:04.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:04.697 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:04.695+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:04.798 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:04.796+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:04.828 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:04.828 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:04.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.095638207999997, 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.829 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:04.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:30:04.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.652746886000003, 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.835 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:30:04.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 78.139288858, 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:30:04.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:04.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 78.129199591000003, 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:30:04.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:04.852 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.850486564000001, 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:04.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:30:04.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:30:04.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:04.859 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 49.902327591000002, 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:04.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:30:04.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:30:04.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:04.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:30:04.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:30:04.866 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815147606, 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.867 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:04.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:30:04.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:30:04.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815130008000001, 2024-09-16T13:30:04.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:30:04.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:30:04.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:30:04.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:04.877 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815115308999999, 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:30:04.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815104378000001, 2024-09-16T13:30:04.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:30:04.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815088758999998, 2024-09-16T13:30:04.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:30:04.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815067496000001, 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.893 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.897 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815052485999999, 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:30:04.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:30:04.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.815036405000001, 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.902 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:30:04.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.906 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.814996537999999, 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.907 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:30:04.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:30:04.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:30:04.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.814985186000001, 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.911 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:30:04.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:30:04.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.090434299999998, 2024-09-16T13:30:04.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:30:04.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:30:04.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.089145974000001, 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:30:04.923 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:04.924 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:30:04.925 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.088874039, 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.926 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.927 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:30:04.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:30:04.929 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.083395416999998, 2024-09-16T13:30:04.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.931 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.932 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.933 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:04.934 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:04.935 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:30:04.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:04.937 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:04.934+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:05.036 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:05.034+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:109499 4.a 4.53ab886a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:30:05.038247+0000", 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.8267e-05, 2024-09-16T13:30:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7947000000000001e-05, 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "queued for pg", 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 109499 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038247+0000", 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038247+0000", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038248+0000", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3799999999999997e-07 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038251+0000", 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2480000000000001e-06 2024-09-16T13:30:05.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038260+0000", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6419999999999996e-06 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038263+0000", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2760000000000001e-06 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:30:05.038297+0000", 2024-09-16T13:30:05.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4068e-05 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 1 2024-09-16T13:30:05.043 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:05.137 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:05.136+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:05.187 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.188 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:05.188 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:05.188 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:05.188 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:05.238 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:05.236+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.646352253999993, 2024-09-16T13:30:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:30:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.636780775999995, 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.789351835000005, 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:30:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043084675000003, 2024-09-16T13:30:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:30:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:30:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043076933999998, 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:30:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043068375000004, 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043059498000005, 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043045633000006, 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:30:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:30:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043037166000005, 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:30:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:30:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.043028363000005, 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:30:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:30:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 74.419558304999995, 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:30:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:30:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 74.406329623999994, 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:30:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:30:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:30:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.892503435000002, 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:30:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:30:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:30:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.875447557000001, 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:30:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:30:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.971296977000002, 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:30:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.960657937000001, 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:30:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100149 4.1f 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.601184785000001, 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037897414999999997, 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100149 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732188+0000", 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8050000000000001e-06 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734548+0000", 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002359758 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734560+0000", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1919999999999999e-05 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734569+0000", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0620000000000007e-06 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734606+0000", 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6939e-05 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734726+0000", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119768 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735030+0000", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000304109 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765403+0000", 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030373052000000001 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.769969+0000", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045659089999999999 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770038+0000", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8748e-05 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770084+0000", 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6344999999999998e-05 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100148 4.12 e95/32 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head v 95'622, mlcod=92'620)", 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.598651517, 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037682625999999997, 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737443+0000", 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027229519999999998 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737460+0000", 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6569e-05 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737471+0000", 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1401e-05 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737528+0000", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7046999999999998e-05 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737551+0000", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2459e-05 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772360+0000", 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034809253999999998 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772402+0000", 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2459999999999997e-05 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100332 4.1f 4:f86ca47e:::rbd_data.11a97e9c0852.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.307970015999999, 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054467106000000001, 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100332 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025403+0000", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6330000000000001e-06 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025408+0000", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9049999999999996e-06 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025420+0000", 2024-09-16T13:30:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2029e-05 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025424+0000", 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4819999999999996e-06 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025459+0000", 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4489000000000001e-05 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025514+0000", 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4737000000000001e-05 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.028379+0000", 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028655379999999999 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079727+0000", 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051347589999999999 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079800+0000", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3681000000000001e-05 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079840+0000", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9243999999999999e-05 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079868+0000", 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:100332 4.1f e95/32 ondisk, result = 0)", 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 23.297202367000001, 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043713039000000002, 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036170+0000", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4399999999999999e-07 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036173+0000", 2024-09-16T13:30:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036182+0000", 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3689999999999996e-06 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036187+0000", 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2900000000000002e-06 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079788+0000", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043600379000000002 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079796+0000", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9270000000000005e-06 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079882+0000", 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6323999999999995e-05 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.430 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:05.431 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:05.429+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:05.532 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:05.530+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:05.569 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:05.569 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:05.569 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:05.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:05.667 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:05.665+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:05.674 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.674 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:05.674 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:05.675 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:05.675 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:05.733 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:05.731+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:05.740 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:05.740 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:05.755 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:05.756 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:05.770 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:05.770 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 120.490900006, 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.783 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.78898107099999, 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:30:05.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.795 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.788958562, 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:30:05.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.801 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.788912235, 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:30:05.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:30:05.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.788745478, 2024-09-16T13:30:05.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:30:05.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:30:05.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.78873071300001, 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.813 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:30:05.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.443408396, 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:30:05.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:30:05.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.824 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.443370718, 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:30:05.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:05.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:30:05.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.443347905, 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:30:05.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:30:05.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:30:05.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.836 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 119.443262692, 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:30:05.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:30:05.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.136683766999994, 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:30:05.842 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.846 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.110775532999995, 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:30:05.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.042010378000001, 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:30:05.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:30:05.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 56.540467561, 2024-09-16T13:30:05.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:30:05.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:30:05.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.345542242, 2024-09-16T13:30:05.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:30:05.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:30:05.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.189665206999997, 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:30:05.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:30:05.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:30:05.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.910713377, 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:30:05.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.135358669999999, 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:30:05.872 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:30:05.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:30:05.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.876 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.038088249000001, 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:30:05.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.036406996, 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:05.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:30:05.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:05.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:05.885 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:05.886 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:05.887 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:05.887 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:05.887 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:05.888 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:05.888 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:05.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:05.889 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:05.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:05.889 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:05.890 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:05.890 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:05.885+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:05.988 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:05.986+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:06.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:30:06.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:30:06.001 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T13:30:06.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-09-16T13:30:06.046 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.046 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:06.089 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:06.087+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:06.190 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:06.188+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:06.198 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.198 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 90 (0%) 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:avg 11.25 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 3.15238 -> 3.15238 (expected baseline 3.13748) 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 7 -> 7 pgs (0.622222 -> 0.622222 * mean) 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 18 -> 18 pgs (1.6 -> 1.6 * mean) 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0033 2024-09-16T13:30:06.284 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0334 2024-09-16T13:30:06.285 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2024-09-16T13:30:06.291 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:06.289+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:06.297 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T13:30:06.297 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-09-16T13:30:06.348 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.348 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:06.392 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:06.390+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:06.493 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:06.491+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:06.498 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.499 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:06.595 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:06.593+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:06.607 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 90 (0%) 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:avg 11.25 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 3.15238 -> 3.15238 (expected baseline 3.13748) 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 7 -> 7 pgs (0.622222 -> 0.622222 * mean) 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 18 -> 18 pgs (1.6 -> 1.6 * mean) 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0033 2024-09-16T13:30:06.608 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0334 2024-09-16T13:30:06.609 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2024-09-16T13:30:06.609 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.609 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:06.621 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T13:30:06.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-16T13:30:06.696 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:06.694+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:06.718 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.718 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:06.797 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:06.795+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:06.829 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.829 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:06.922 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:06.921+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:06.931 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 90 (0%) 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:avg 11.25 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 3.15238 -> 3.15238 (expected baseline 3.13748) 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 7 -> 7 pgs (0.622222 -> 0.622222 * mean) 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 18 -> 18 pgs (1.6 -> 1.6 * mean) 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2024-09-16T13:30:06.932 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2024-09-16T13:30:06.933 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0033 2024-09-16T13:30:06.933 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0037 2024-09-16T13:30:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2024-09-16T13:30:06.944 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T13:30:06.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-09-16T13:30:06.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:30:06.986 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:06.986 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:06.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:06.998 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:06.997+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:07.099 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:07.097+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:07.152 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.152 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:07.200 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:07.198+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:07.264 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.264 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:07.301 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:07.299+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:07.321 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 90 (0%) 2024-09-16T13:30:07.321 INFO:teuthology.orchestra.run.smithi076.stdout:avg 11.25 2024-09-16T13:30:07.321 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 3.15238 -> 3.15238 (expected baseline 3.13748) 2024-09-16T13:30:07.321 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 7 -> 7 pgs (0.622222 -> 0.622222 * mean) 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 18 -> 18 pgs (1.6 -> 1.6 * mean) 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0033 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0334 2024-09-16T13:30:07.322 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2024-09-16T13:30:07.334 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T13:30:07.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T13:30:07.374 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.374 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:07.401 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:07.400+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:07.502 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:07.500+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:07.523 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.523 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:07.603 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:07.602+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 90 (0%) 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:avg 11.25 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 3.15238 -> 3.15238 (expected baseline 3.13748) 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 7 -> 7 pgs (0.622222 -> 0.622222 * mean) 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 18 -> 18 pgs (1.6 -> 1.6 * mean) 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2024-09-16T13:30:07.604 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2024-09-16T13:30:07.605 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2024-09-16T13:30:07.605 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0033 2024-09-16T13:30:07.605 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0037 2024-09-16T13:30:07.605 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2024-09-16T13:30:07.634 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.634 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:07.704 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:07.702+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:07.741 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.741 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:07.833 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:07.831+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:07.887 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:07.887 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:07.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:07.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:07.936 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:07.934+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:07.959 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:30:08.006 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:08.004+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:08.022 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:08.023 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:08.039 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:08.040 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:08.049 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:08.049 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:08.057 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:08.057 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:08.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:08.073 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:08.090 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:08.106 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:08.107 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:08.105+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:08.123 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:08.139 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:08.139 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:08.199 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:08.199 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:08.199 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:08.199 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:08.199 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:08.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:08.208 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:08.205+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:08.309 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:08.307+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:08.348 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 93.615989647000006, 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.349 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:30:08.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.173098324999998, 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.355 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:30:08.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 81.659640296999996, 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:30:08.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:30:08.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:08.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.365 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 81.649551029999998, 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:08.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:08.369 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.370838003000003, 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.372 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:30:08.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.374 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 53.422679029999998, 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.376 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:30:08.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:30:08.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335499044999999, 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.382 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:30:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:08.386 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335481446999999, 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:30:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335466748000002, 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:30:08.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335455817000003, 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.395 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335440198000001, 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:30:08.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335418935, 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.404 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:08.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335403925000001, 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:30:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335387844000003, 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:30:08.413 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:30:08.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335347976999998, 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:30:08.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 34.335336624999997, 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.422 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:08.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:30:08.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.610785739000001, 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:30:08.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:30:08.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:30:08.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.609497413, 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:30:08.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.609225477999999, 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.603746856000001, 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.441 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.445 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:08.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:08.447 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:08.445+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:08.548 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:08.546+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:08.587 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:08.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:08.587 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:08.587 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:08.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:08.649 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:08.647+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:08.694 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:08.694 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:08.694 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:08.694 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:08.694 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:08.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:08.750 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:08.748+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:08.802 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 95.112849260000004, 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.803 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:08.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:30:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:30:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 95.103277782000006, 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:30:08.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.255848841000002, 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:30:08.813 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509581681, 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509573939999996, 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:30:08.822 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:30:08.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509565381000002, 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.827 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:08.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:30:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509556504000003, 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:30:08.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:30:08.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:30:08.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.835 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509542639000003, 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:30:08.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509534172000002, 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.840 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:30:08.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.509525369000002, 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:30:08.844 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:30:08.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.848 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.886055311000007, 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:30:08.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:30:08.854 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.872826630000006, 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:30:08.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:08.858 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 67.359000441000006, 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 67.341944562999998, 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:30:08.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.437793982999999, 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:30:08.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:30:08.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:08.874 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.427154942999998, 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100149 4.1f 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:08.878 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.067681790999998, 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037897414999999997, 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100149 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732188+0000", 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8050000000000001e-06 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734548+0000", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002359758 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734560+0000", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1919999999999999e-05 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734569+0000", 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0620000000000007e-06 2024-09-16T13:30:08.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734606+0000", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6939e-05 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734726+0000", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119768 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735030+0000", 2024-09-16T13:30:08.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000304109 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765403+0000", 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030373052000000001 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.769969+0000", 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045659089999999999 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770038+0000", 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8748e-05 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770084+0000", 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6344999999999998e-05 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100148 4.12 e95/32 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head v 95'622, mlcod=92'620)", 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:08.884 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.065148523000001, 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037682625999999997, 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737443+0000", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027229519999999998 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737460+0000", 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6569e-05 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737471+0000", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1401e-05 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737528+0000", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7046999999999998e-05 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737551+0000", 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2459e-05 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772360+0000", 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034809253999999998 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772402+0000", 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2459999999999997e-05 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100332 4.1f 4:f86ca47e:::rbd_data.11a97e9c0852.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:08.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.774467022, 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054467106000000001, 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100332 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025403+0000", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6330000000000001e-06 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025408+0000", 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9049999999999996e-06 2024-09-16T13:30:08.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025420+0000", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2029e-05 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025424+0000", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4819999999999996e-06 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025459+0000", 2024-09-16T13:30:08.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4489000000000001e-05 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025514+0000", 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4737000000000001e-05 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.028379+0000", 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028655379999999999 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:08.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079727+0000", 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051347589999999999 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079800+0000", 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3681000000000001e-05 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079840+0000", 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9243999999999999e-05 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079868+0000", 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:100332 4.1f e95/32 ondisk, result = 0)", 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 26.763699373000001, 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043713039000000002, 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036170+0000", 2024-09-16T13:30:08.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4399999999999999e-07 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036173+0000", 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036182+0000", 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3689999999999996e-06 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036187+0000", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2900000000000002e-06 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079788+0000", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043600379000000002 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079796+0000", 2024-09-16T13:30:08.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9270000000000005e-06 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079882+0000", 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6323999999999995e-05 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:08.898 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:08.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:08.900 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:08.898+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:09.001 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:08.999+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:09.049 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:09.050 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:09.050 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:09.050 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:09.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:09.102 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:09.100+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:09.159 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:09.159 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:09.159 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:09.159 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:09.160 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:09.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:09.203 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:09.202+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:09.304 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:09.302+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:09.308 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:09.308 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:09.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:09.308 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:09.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 124.016238997, 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:30:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:09.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.31432006199999, 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:30:09.316 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:09.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:30:09.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:30:09.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.314297553, 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:09.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.314251226, 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:30:09.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:30:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:30:09.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.31408446899999, 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.336 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:09.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:30:09.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.314069704, 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.343 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:09.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:30:09.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:30:09.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 122.96874738699999, 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:30:09.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:30:09.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:30:09.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 122.968709709, 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:30:09.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:09.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:09.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:30:09.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:30:09.363 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 122.96868689599999, 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.364 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:30:09.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:30:09.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:30:09.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 122.968601683, 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:09.370 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.372 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:09.373 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.376 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:09.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:09.378 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.379 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 94.662022758000006, 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.383 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 94.636114524000007, 2024-09-16T13:30:09.386 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:30:09.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 94.567349368999999, 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.391 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:30:09.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.394 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:30:09.395 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.065806551999998, 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.396 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:30:09.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.398 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.399 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.870881232999999, 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:30:09.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.401 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:30:09.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:30:09.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 36.715004198000003, 2024-09-16T13:30:09.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:30:09.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.409 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.409 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:30:09.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:30:09.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:30:09.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:30:09.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:30:09.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:30:09.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.436052367999999, 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.419 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.420 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:09.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:30:09.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:30:09.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:30:09.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:30:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:30:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:30:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:30:09.429 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.660697661, 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:09.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:30:09.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.563427239999999, 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:30:09.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 27.561745986999998, 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:30:09.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:30:09.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:09.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:09.445 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:09.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:09.447 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:09.445+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:09.547 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:09.545+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:09.547 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:09.547 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:09.548 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:09.548 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:09.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:09.648 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:09.646+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:09.694 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:09.694 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:09.694 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:09.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:09.749 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:09.747+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:09.802 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:09.802 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 92.633419415999995, 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:30:09.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:30:09.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:30:09.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:30:09.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 92.627943850999998, 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:30:09.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.867617043999999, 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:30:09.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:30:09.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.851078919000003, 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:30:09.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.824 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.508375143999999, 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.508359339999998, 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:30:09.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:09.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:30:09.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:30:09.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.508343779, 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.834 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:30:09.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.839 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.508326065000006, 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:30:09.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:30:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.508299481999998, 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:30:09.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.666745569, 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:30:09.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.469305997999996, 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:30:09.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.894429392000006, 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:30:09.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:30:09.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.547139434000002, 2024-09-16T13:30:09.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:09.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:09.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:30:09.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:30:09.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:30:09.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:30:09.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.546859538, 2024-09-16T13:30:09.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:30:09.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:30:09.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:30:09.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 56.115167821999997, 2024-09-16T13:30:09.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.887 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:30:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:30:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:30:09.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.521096440000001, 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:30:09.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:30:09.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:30:09.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.452366738999999, 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.901 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:30:09.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:30:09.906 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 42.441502, 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:30:09.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:30:09.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100190 4.1e 4:7a9178f5:::rbd_data.11a97e9c0852.0000000000000012:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.997418206999999, 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048355217999999998, 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.911 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100190 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801667+0000", 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801670+0000", 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801678+0000", 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3119999999999996e-06 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801681+0000", 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801710+0000", 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9496999999999999e-05 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801752+0000", 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1288000000000002e-05 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.804108+0000", 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023560870000000002 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849951+0000", 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045842898999999999 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849959+0000", 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5180000000000004e-06 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849995+0000", 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6241000000000001e-05 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.850022+0000", 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6336999999999999e-05 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103827 4.1e 4:7b9e4072:::rbd_data.11a976e61a66.0000000000000020:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.606372775000001, 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046988757999999999, 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:09.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103827 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192713+0000", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9000000000000005e-07 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192718+0000", 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7060000000000003e-06 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192728+0000", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0730999999999999e-05 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192733+0000", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6079999999999998e-06 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192774+0000", 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0859999999999998e-05 2024-09-16T13:30:09.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192861+0000", 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7551000000000003e-05 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.195707+0000", 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028459779999999999 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:09.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239591+0000", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043883414000000003 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239603+0000", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1843e-05 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239660+0000", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7012000000000002e-05 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:09.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239701+0000", 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1065000000000001e-05 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:09.923 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:09.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:09.925 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:09.922+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:09.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:30:10.052 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:10.050+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:10.073 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:10.074 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:10.074 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:10.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:10.126 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:10.124+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:10.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:10.224 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:10.224 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:10.224 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:10.226 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:10.224+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:10.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:10.283 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:10.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:10.300 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:10.316 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:10.327 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:10.325+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:10.331 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:10.331 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:10.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:10.347 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:10.363 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:10.363 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:10.379 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:10.379 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:10.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:10.396 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:10.447 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:10.445+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:10.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:10.540 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:10.538+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:10.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:10.644 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:10.642+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:10.731 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:10.729+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:10.741 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:10.741 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:10.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:10.833 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:10.831+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:10.889 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:10.889 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:10.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:10.941 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:10.940+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:10.975 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:30:11.008 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.008 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.015 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:11.057 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:11.055+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:11.136 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:11.134+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:11.162 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.162 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:11.237 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:11.235+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:11.276 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.277 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:11.370 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:11.368+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:11.423 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.424 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:11.438 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:11.436+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:11.539 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:11.538+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:11.571 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.571 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:11.640 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:11.639+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:11.722 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.723 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:11.775 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:11.773+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:11.830 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.830 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:11.842 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:11.840+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:11.943 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:11.941+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:11.978 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:11.978 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:11.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:12.043 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:12.041+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:12.132 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:12.133 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:12.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:12.144 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:12.142+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:12.241 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:12.241 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:12.244 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:12.242+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:12.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:12.346 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:12.344+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:12.388 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:12.388 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:12.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:12.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:12.446 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:12.444+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:12.495 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:12.495 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:12.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:12.523 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:12.523 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:12.538 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:12.538 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:12.547 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:12.545+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:12.552 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:12.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:12.567 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:12.583 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:12.602 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:12.602 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:12.617 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:12.617 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:12.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:30:12.618 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:30:12.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:12.633 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:12.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:12.644 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:12.644 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:12.644 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:12.645 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:12.645 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:12.647 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:12.646+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:12.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:12.749 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:12.747+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 127.50178036, 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:30:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.799861425, 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:30:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:30:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:30:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:30:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.799838916, 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:30:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:30:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:30:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:30:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.79979258900001, 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:30:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.799625832, 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:30:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.799611067, 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:30:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.45428875, 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.45425107200001, 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:30:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.454228259, 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:30:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:30:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 126.454143046, 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:30:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 98.147564121000002, 2024-09-16T13:30:12.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:30:12.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.873 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:30:12.874 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 98.121655887000003, 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.876 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:30:12.877 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:12.879 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 98.052890731999995, 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.880 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:30:12.881 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.882 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:30:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.551347915000001, 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:12.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.885 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:30:12.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.887 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 49.356422596000002, 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.888 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.889 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.890 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.891 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:30:12.892 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.200545560999998, 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.894 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.895 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:30:12.896 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.897 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.898 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.899 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 38.921593731000002, 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.900 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:12.901 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:30:12.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.903 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.904 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:30:12.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.146239024, 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.907 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.910 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.911 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:30:12.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:30:12.913 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.048968602999999, 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.914 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.915 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:30:12.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 31.047287350000001, 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:12.918 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:30:12.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:30:12.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:30:12.921 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:12.923 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:12.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:12.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:12.927 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:12.928 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-16T13:30:11.981117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1051900810,"num_objects":1382,"num_object_clones":304,"num_object_copies":2764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1377,"num_whiteouts":556,"num_read":158989,"num_read_kb":798150,"num_write":20418,"num_write_kb":462237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":28,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16450,"ondisk_log_size":16450,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2482104,"kb_used_data":57220,"kb_used_omap":13,"kb_used_meta":2424498,"kb_avail":752492616,"statfs":{"total":773094113280,"available":770552438784,"internally_reserved":0,"allocated":58593280,"data_stored":464421567,"data_compressed":11064145,"data_compressed_allocated":37326848,"data_compressed_original":452320007,"omap_allocated":13632,"internal_metadata":2482686656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-86119676,"num_objects":-312,"num_object_clones":-119,"num_object_copies":-624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-312,"num_whiteouts":-102,"num_read":2681,"num_read_kb":2270,"num_write":260,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026771"},"pg_stats":[{"pgid":"4.1f","version":"120'314","reported_seq":1954,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.965860+0000","last_change":"2024-09-16T13:30:07.966093+0000","last_active":"2024-09-16T13:30:07.965860+0000","last_peered":"2024-09-16T13:30:07.965860+0000","last_clean":"2024-09-16T13:30:07.965860+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:07.965860+0000","last_undegraded":"2024-09-16T13:30:07.965860+0000","last_fullsized":"2024-09-16T13:30:07.965860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0065669020000000003,"stat_sum":{"num_bytes":49063477,"num_objects":41,"num_object_clones":9,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":18,"num_read":2544,"num_read_kb":19874,"num_write":327,"num_write_kb":9426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.1e","version":"120'1011","reported_seq":11497,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.122553+0000","last_change":"2024-09-16T13:30:08.003339+0000","last_active":"2024-09-16T13:30:08.122553+0000","last_peered":"2024-09-16T13:30:08.122553+0000","last_clean":"2024-09-16T13:30:08.122553+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:08.122553+0000","last_undegraded":"2024-09-16T13:30:08.122553+0000","last_fullsized":"2024-09-16T13:30:08.122553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1011,"log_dups_size":0,"ondisk_log_size":1011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0060451680000000001,"stat_sum":{"num_bytes":53367204,"num_objects":39,"num_object_clones":6,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":14,"num_read":14668,"num_read_kb":30798,"num_write":1597,"num_write_kb":9673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.1d","version":"121'393","reported_seq":1386,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.043489+0000","last_change":"2024-09-16T13:30:08.967976+0000","last_active":"2024-09-16T13:30:09.043489+0000","last_peered":"2024-09-16T13:30:09.043489+0000","last_clean":"2024-09-16T13:30:09.043489+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:09.043489+0000","last_undegraded":"2024-09-16T13:30:09.043489+0000","last_fullsized":"2024-09-16T13:30:09.043489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0062369340000000004,"stat_sum":{"num_bytes":34382820,"num_objects":33,"num_object_clones":8,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":18,"num_read":1788,"num_read_kb":7937,"num_write":413,"num_write_kb":5823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.1c","version":"120'406","reported_seq":4009,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.104048+0000","last_change":"2024-09-16T13:30:07.959985+0000","last_active":"2024-09-16T13:30:08.104048+0000","last_peered":"2024-09-16T13:30:08.104048+0000","last_clean":"2024-09-16T13:30:08.104048+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:08.104048+0000","last_undegraded":"2024-09-16T13:30:08.104048+0000","last_fullsized":"2024-09-16T13:30:08.104048+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0040272270000000004,"stat_sum":{"num_bytes":25145986,"num_objects":40,"num_object_clones":10,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":12,"num_read":6890,"num_read_kb":26273,"num_write":417,"num_write_kb":9862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.1b","version":"121'440","reported_seq":2198,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.997646+0000","last_change":"2024-09-16T13:30:08.997646+0000","last_active":"2024-09-16T13:30:08.997646+0000","last_peered":"2024-09-16T13:30:08.997646+0000","last_clean":"2024-09-16T13:30:08.997646+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:08.997646+0000","last_undegraded":"2024-09-16T13:30:08.997646+0000","last_fullsized":"2024-09-16T13:30:08.997646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0090342979999999996,"stat_sum":{"num_bytes":30963251,"num_objects":42,"num_object_clones":11,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":16,"num_read":3240,"num_read_kb":15629,"num_write":521,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.1a","version":"121'380","reported_seq":3958,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.103982+0000","last_change":"2024-09-16T13:30:08.990794+0000","last_active":"2024-09-16T13:30:09.103982+0000","last_peered":"2024-09-16T13:30:09.103982+0000","last_clean":"2024-09-16T13:30:09.103982+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:09.103982+0000","last_undegraded":"2024-09-16T13:30:09.103982+0000","last_fullsized":"2024-09-16T13:30:09.103982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.002216848,"stat_sum":{"num_bytes":28191583,"num_objects":39,"num_object_clones":5,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":12,"num_read":3908,"num_read_kb":16361,"num_write":427,"num_write_kb":6964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.19","version":"121'362","reported_seq":3323,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.067483+0000","last_change":"2024-09-16T13:30:08.973902+0000","last_active":"2024-09-16T13:30:09.067483+0000","last_peered":"2024-09-16T13:30:09.067483+0000","last_clean":"2024-09-16T13:30:09.067483+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:09.067483+0000","last_undegraded":"2024-09-16T13:30:09.067483+0000","last_fullsized":"2024-09-16T13:30:09.067483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011829753,"stat_sum":{"num_bytes":33003259,"num_objects":53,"num_object_clones":11,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":19,"num_read":3411,"num_read_kb":15847,"num_write":374,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.18","version":"121'351","reported_seq":3266,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.989027+0000","last_change":"2024-09-16T13:30:08.989027+0000","last_active":"2024-09-16T13:30:08.989027+0000","last_peered":"2024-09-16T13:30:08.989027+0000","last_clean":"2024-09-16T13:30:08.989027+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:08.989027+0000","last_undegraded":"2024-09-16T13:30:08.989027+0000","last_fullsized":"2024-09-16T13:30:08.989027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050231099999999997,"stat_sum":{"num_bytes":51083880,"num_objects":45,"num_object_clones":13,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":14,"num_read":3436,"num_read_kb":32798,"num_write":377,"num_write_kb":30744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.17","version":"120'335","reported_seq":2888,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.077498+0000","last_change":"2024-09-16T13:30:08.970899+0000","last_active":"2024-09-16T13:30:09.077498+0000","last_peered":"2024-09-16T13:30:09.077498+0000","last_clean":"2024-09-16T13:30:09.077498+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:09.077498+0000","last_undegraded":"2024-09-16T13:30:09.077498+0000","last_fullsized":"2024-09-16T13:30:09.077498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086546460000000002,"stat_sum":{"num_bytes":29288747,"num_objects":50,"num_object_clones":11,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":26,"num_read":4075,"num_read_kb":29446,"num_write":320,"num_write_kb":17559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.16","version":"118'585","reported_seq":2582,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.002416+0000","last_change":"2024-09-16T13:30:08.002686+0000","last_active":"2024-09-16T13:30:08.002416+0000","last_peered":"2024-09-16T13:30:08.002416+0000","last_clean":"2024-09-16T13:30:08.002416+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:08.002416+0000","last_undegraded":"2024-09-16T13:30:08.002416+0000","last_fullsized":"2024-09-16T13:30:08.002416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053928270000000002,"stat_sum":{"num_bytes":11948411,"num_objects":34,"num_object_clones":6,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":12,"num_read":4420,"num_read_kb":15745,"num_write":739,"num_write_kb":9974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.15","version":"121'775","reported_seq":3945,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.968559+0000","last_change":"2024-09-16T13:30:08.968732+0000","last_active":"2024-09-16T13:30:08.968559+0000","last_peered":"2024-09-16T13:30:08.968559+0000","last_clean":"2024-09-16T13:30:08.968559+0000","last_became_active":"2024-09-16T13:29:55.466429+0000","last_became_peered":"2024-09-16T13:29:55.466429+0000","last_unstale":"2024-09-16T13:30:08.968559+0000","last_undegraded":"2024-09-16T13:30:08.968559+0000","last_fullsized":"2024-09-16T13:30:08.968559+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:31.786154+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0043221739999999998,"stat_sum":{"num_bytes":65744670,"num_objects":46,"num_object_clones":11,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":18,"num_read":5752,"num_read_kb":34328,"num_write":1185,"num_write_kb":18012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.14","version":"121'435","reported_seq":3239,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.968775+0000","last_change":"2024-09-16T13:30:08.968775+0000","last_active":"2024-09-16T13:30:08.968775+0000","last_peered":"2024-09-16T13:30:08.968775+0000","last_clean":"2024-09-16T13:30:08.968775+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:08.968775+0000","last_undegraded":"2024-09-16T13:30:08.968775+0000","last_fullsized":"2024-09-16T13:30:08.968775+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067900699999999996,"stat_sum":{"num_bytes":8611380,"num_objects":42,"num_object_clones":10,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":20,"num_read":4742,"num_read_kb":20326,"num_write":476,"num_write_kb":8921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.13","version":"121'337","reported_seq":1839,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.995668+0000","last_change":"2024-09-16T13:30:08.996032+0000","last_active":"2024-09-16T13:30:08.995668+0000","last_peered":"2024-09-16T13:30:08.995668+0000","last_clean":"2024-09-16T13:30:08.995668+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:08.995668+0000","last_undegraded":"2024-09-16T13:30:08.995668+0000","last_fullsized":"2024-09-16T13:30:08.995668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0071533270000000001,"stat_sum":{"num_bytes":40357359,"num_objects":50,"num_object_clones":13,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":23,"num_read":1518,"num_read_kb":22419,"num_write":336,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.12","version":"120'755","reported_seq":3944,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.087962+0000","last_change":"2024-09-16T13:30:07.964650+0000","last_active":"2024-09-16T13:30:08.087962+0000","last_peered":"2024-09-16T13:30:08.087962+0000","last_clean":"2024-09-16T13:30:08.087962+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:08.087962+0000","last_undegraded":"2024-09-16T13:30:08.087962+0000","last_fullsized":"2024-09-16T13:30:08.087962+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0076586320000000003,"stat_sum":{"num_bytes":5749945,"num_objects":31,"num_object_clones":3,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":13,"num_read":8401,"num_read_kb":20371,"num_write":936,"num_write_kb":8817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.11","version":"121'728","reported_seq":4659,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.965230+0000","last_change":"2024-09-16T13:30:08.965230+0000","last_active":"2024-09-16T13:30:08.965230+0000","last_peered":"2024-09-16T13:30:08.965230+0000","last_clean":"2024-09-16T13:30:08.965230+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:08.965230+0000","last_undegraded":"2024-09-16T13:30:08.965230+0000","last_fullsized":"2024-09-16T13:30:08.965230+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0033193480000000002,"stat_sum":{"num_bytes":12036586,"num_objects":45,"num_object_clones":12,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":20,"num_read":5901,"num_read_kb":16214,"num_write":934,"num_write_kb":25099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.10","version":"121'676","reported_seq":3476,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.104749+0000","last_change":"2024-09-16T13:30:08.969742+0000","last_active":"2024-09-16T13:30:09.104749+0000","last_peered":"2024-09-16T13:30:09.104749+0000","last_clean":"2024-09-16T13:30:09.104749+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:09.104749+0000","last_undegraded":"2024-09-16T13:30:09.104749+0000","last_fullsized":"2024-09-16T13:30:09.104749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0078094059999999996,"stat_sum":{"num_bytes":8722690,"num_objects":30,"num_object_clones":5,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":15,"num_read":7763,"num_read_kb":20746,"num_write":939,"num_write_kb":9520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"2.7","version":"0'0","reported_seq":130,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.960925+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:06.960925+0000","last_peered":"2024-09-16T13:30:06.960925+0000","last_clean":"2024-09-16T13:30:06.960925+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:06.960925+0000","last_undegraded":"2024-09-16T13:30:06.960925+0000","last_fullsized":"2024-09-16T13:30:06.960925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"121'453","reported_seq":3427,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.048819+0000","last_change":"2024-09-16T13:30:08.993686+0000","last_active":"2024-09-16T13:30:09.048819+0000","last_peered":"2024-09-16T13:30:09.048819+0000","last_clean":"2024-09-16T13:30:09.048819+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:09.048819+0000","last_undegraded":"2024-09-16T13:30:09.048819+0000","last_fullsized":"2024-09-16T13:30:09.048819+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.0051290820000000001,"stat_sum":{"num_bytes":19144426,"num_objects":45,"num_object_clones":12,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":20,"num_read":3486,"num_read_kb":37810,"num_write":567,"num_write_kb":17612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.954066+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:06.954066+0000","last_peered":"2024-09-16T13:30:06.954066+0000","last_clean":"2024-09-16T13:30:06.954066+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:06.954066+0000","last_undegraded":"2024-09-16T13:30:06.954066+0000","last_fullsized":"2024-09-16T13:30:06.954066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"121'312","reported_seq":2455,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.048485+0000","last_change":"2024-09-16T13:30:08.968397+0000","last_active":"2024-09-16T13:30:09.048485+0000","last_peered":"2024-09-16T13:30:09.048485+0000","last_clean":"2024-09-16T13:30:09.048485+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:09.048485+0000","last_undegraded":"2024-09-16T13:30:09.048485+0000","last_fullsized":"2024-09-16T13:30:09.048485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064356379999999996,"stat_sum":{"num_bytes":13407144,"num_objects":46,"num_object_clones":10,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":19,"num_read":2284,"num_read_kb":15218,"num_write":400,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.960198+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:06.960198+0000","last_peered":"2024-09-16T13:30:06.960198+0000","last_clean":"2024-09-16T13:30:06.960198+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:06.960198+0000","last_undegraded":"2024-09-16T13:30:06.960198+0000","last_fullsized":"2024-09-16T13:30:06.960198+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'662","reported_seq":3834,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.064233+0000","last_change":"2024-09-16T13:30:07.999123+0000","last_active":"2024-09-16T13:30:08.064233+0000","last_peered":"2024-09-16T13:30:08.064233+0000","last_clean":"2024-09-16T13:30:08.064233+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:08.064233+0000","last_undegraded":"2024-09-16T13:30:08.064233+0000","last_fullsized":"2024-09-16T13:30:08.064233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001895433,"stat_sum":{"num_bytes":50455096,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":21,"num_read":4847,"num_read_kb":49449,"num_write":767,"num_write_kb":31496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.953793+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:06.953793+0000","last_peered":"2024-09-16T13:30:06.953793+0000","last_clean":"2024-09-16T13:30:06.953793+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:06.953793+0000","last_undegraded":"2024-09-16T13:30:06.953793+0000","last_fullsized":"2024-09-16T13:30:06.953793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"120'537","reported_seq":3194,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.093244+0000","last_change":"2024-09-16T13:30:07.961332+0000","last_active":"2024-09-16T13:30:08.093244+0000","last_peered":"2024-09-16T13:30:08.093244+0000","last_clean":"2024-09-16T13:30:08.093244+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:08.093244+0000","last_undegraded":"2024-09-16T13:30:08.093244+0000","last_fullsized":"2024-09-16T13:30:08.093244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.004204625,"stat_sum":{"num_bytes":10481799,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":8,"num_read":6088,"num_read_kb":18139,"num_write":618,"num_write_kb":26206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.954626+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:07.954626+0000","last_peered":"2024-09-16T13:30:07.954626+0000","last_clean":"2024-09-16T13:30:07.954626+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:07.954626+0000","last_undegraded":"2024-09-16T13:30:07.954626+0000","last_fullsized":"2024-09-16T13:30:07.954626+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":38,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.954523+0000","last_change":"2024-09-16T13:29:55.469146+0000","last_active":"2024-09-16T13:30:06.954523+0000","last_peered":"2024-09-16T13:30:06.954523+0000","last_clean":"2024-09-16T13:30:06.954523+0000","last_became_active":"2024-09-16T13:29:55.468831+0000","last_became_peered":"2024-09-16T13:29:55.468831+0000","last_unstale":"2024-09-16T13:30:06.954523+0000","last_undegraded":"2024-09-16T13:30:06.954523+0000","last_fullsized":"2024-09-16T13:30:06.954523+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:56.220967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"120'341","reported_seq":4384,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.975662+0000","last_change":"2024-09-16T13:30:07.976021+0000","last_active":"2024-09-16T13:30:07.975662+0000","last_peered":"2024-09-16T13:30:07.975662+0000","last_clean":"2024-09-16T13:30:07.975662+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:07.975662+0000","last_undegraded":"2024-09-16T13:30:07.975662+0000","last_fullsized":"2024-09-16T13:30:07.975662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.018697760000000001,"stat_sum":{"num_bytes":26513422,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":10,"num_read":4241,"num_read_kb":35943,"num_write":381,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"3.0","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.953845+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:06.953845+0000","last_peered":"2024-09-16T13:30:06.953845+0000","last_clean":"2024-09-16T13:30:06.953845+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:06.953845+0000","last_undegraded":"2024-09-16T13:30:06.953845+0000","last_fullsized":"2024-09-16T13:30:06.953845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.962586+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:06.962586+0000","last_peered":"2024-09-16T13:30:06.962586+0000","last_clean":"2024-09-16T13:30:06.962586+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:06.962586+0000","last_undegraded":"2024-09-16T13:30:06.962586+0000","last_fullsized":"2024-09-16T13:30:06.962586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"120'422","reported_seq":3200,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.124202+0000","last_change":"2024-09-16T13:30:07.975848+0000","last_active":"2024-09-16T13:30:08.124202+0000","last_peered":"2024-09-16T13:30:08.124202+0000","last_clean":"2024-09-16T13:30:08.124202+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:08.124202+0000","last_undegraded":"2024-09-16T13:30:08.124202+0000","last_fullsized":"2024-09-16T13:30:08.124202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.018103083999999998,"stat_sum":{"num_bytes":52192259,"num_objects":52,"num_object_clones":14,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":19,"num_read":4469,"num_read_kb":20326,"num_write":480,"num_write_kb":8580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"3.1","version":"105'2","reported_seq":105,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.953343+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:06.953343+0000","last_peered":"2024-09-16T13:30:06.953343+0000","last_clean":"2024-09-16T13:30:06.953343+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:06.953343+0000","last_undegraded":"2024-09-16T13:30:06.953343+0000","last_fullsized":"2024-09-16T13:30:06.953343+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.966977+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:07.966977+0000","last_peered":"2024-09-16T13:30:07.966977+0000","last_clean":"2024-09-16T13:30:07.966977+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:07.966977+0000","last_undegraded":"2024-09-16T13:30:07.966977+0000","last_fullsized":"2024-09-16T13:30:07.966977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"120'487","reported_seq":3614,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.044821+0000","last_change":"2024-09-16T13:30:08.007894+0000","last_active":"2024-09-16T13:30:08.044821+0000","last_peered":"2024-09-16T13:30:08.044821+0000","last_clean":"2024-09-16T13:30:08.044821+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:08.044821+0000","last_undegraded":"2024-09-16T13:30:08.044821+0000","last_fullsized":"2024-09-16T13:30:08.044821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0.010483541000000001,"stat_sum":{"num_bytes":52169979,"num_objects":46,"num_object_clones":7,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":21,"num_read":4993,"num_read_kb":40577,"num_write":487,"num_write_kb":30169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"3.2","version":"101'3","reported_seq":107,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.960936+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:07.960936+0000","last_peered":"2024-09-16T13:30:07.960936+0000","last_clean":"2024-09-16T13:30:07.960936+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:07.960936+0000","last_undegraded":"2024-09-16T13:30:07.960936+0000","last_fullsized":"2024-09-16T13:30:07.960936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":85,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":29,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-16T13:30:06.962668+0000","last_change":"2024-09-16T13:29:55.466745+0000","last_active":"2024-09-16T13:30:06.962668+0000","last_peered":"2024-09-16T13:30:06.962668+0000","last_clean":"2024-09-16T13:30:06.962668+0000","last_became_active":"2024-09-16T13:29:55.466418+0000","last_became_peered":"2024-09-16T13:29:55.466418+0000","last_unstale":"2024-09-16T13:30:06.962668+0000","last_undegraded":"2024-09-16T13:30:06.962668+0000","last_fullsized":"2024-09-16T13:30:06.962668+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:33.757459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":68,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.966896+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:07.966896+0000","last_peered":"2024-09-16T13:30:07.966896+0000","last_clean":"2024-09-16T13:30:07.966896+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:07.966896+0000","last_undegraded":"2024-09-16T13:30:07.966896+0000","last_fullsized":"2024-09-16T13:30:07.966896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"118'533","reported_seq":2212,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.966326+0000","last_change":"2024-09-16T13:30:07.966518+0000","last_active":"2024-09-16T13:30:07.966326+0000","last_peered":"2024-09-16T13:30:07.966326+0000","last_clean":"2024-09-16T13:30:07.966326+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:07.966326+0000","last_undegraded":"2024-09-16T13:30:07.966326+0000","last_fullsized":"2024-09-16T13:30:07.966326+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068615150000000003,"stat_sum":{"num_bytes":25363306,"num_objects":43,"num_object_clones":7,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":15,"num_read":3242,"num_read_kb":17043,"num_write":679,"num_write_kb":7420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.8","version":"121'1177","reported_seq":5384,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.971453+0000","last_change":"2024-09-16T13:30:08.971453+0000","last_active":"2024-09-16T13:30:08.971453+0000","last_peered":"2024-09-16T13:30:08.971453+0000","last_clean":"2024-09-16T13:30:08.971453+0000","last_became_active":"2024-09-16T13:29:55.466349+0000","last_became_peered":"2024-09-16T13:29:55.466349+0000","last_unstale":"2024-09-16T13:30:08.971453+0000","last_undegraded":"2024-09-16T13:30:08.971453+0000","last_fullsized":"2024-09-16T13:30:08.971453+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:08.619344+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0070275520000000003,"stat_sum":{"num_bytes":32832870,"num_objects":42,"num_object_clones":10,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":12,"num_read":9240,"num_read_kb":25312,"num_write":1772,"num_write_kb":11383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.9","version":"121'330","reported_seq":4552,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.117339+0000","last_change":"2024-09-16T13:30:08.988988+0000","last_active":"2024-09-16T13:30:09.117339+0000","last_peered":"2024-09-16T13:30:09.117339+0000","last_clean":"2024-09-16T13:30:09.117339+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:09.117339+0000","last_undegraded":"2024-09-16T13:30:09.117339+0000","last_fullsized":"2024-09-16T13:30:09.117339+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043930100000000001,"stat_sum":{"num_bytes":68393781,"num_objects":42,"num_object_clones":13,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":14,"num_read":5716,"num_read_kb":29951,"num_write":405,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.a","version":"120'483","reported_seq":4272,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.967610+0000","last_change":"2024-09-16T13:30:07.967610+0000","last_active":"2024-09-16T13:30:07.967610+0000","last_peered":"2024-09-16T13:30:07.967610+0000","last_clean":"2024-09-16T13:30:07.967610+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:07.967610+0000","last_undegraded":"2024-09-16T13:30:07.967610+0000","last_fullsized":"2024-09-16T13:30:07.967610+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0080573559999999999,"stat_sum":{"num_bytes":44719040,"num_objects":45,"num_object_clones":10,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":17,"num_read":5762,"num_read_kb":43442,"num_write":523,"num_write_kb":25109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.b","version":"120'392","reported_seq":5310,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.999802+0000","last_change":"2024-09-16T13:30:07.999802+0000","last_active":"2024-09-16T13:30:07.999802+0000","last_peered":"2024-09-16T13:30:07.999802+0000","last_clean":"2024-09-16T13:30:07.999802+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:30:07.999802+0000","last_undegraded":"2024-09-16T13:30:07.999802+0000","last_fullsized":"2024-09-16T13:30:07.999802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.00257325,"stat_sum":{"num_bytes":56197597,"num_objects":41,"num_object_clones":9,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":17,"num_read":5827,"num_read_kb":40469,"num_write":479,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.c","version":"121'427","reported_seq":5167,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.994101+0000","last_change":"2024-09-16T13:30:08.994101+0000","last_active":"2024-09-16T13:30:08.994101+0000","last_peered":"2024-09-16T13:30:08.994101+0000","last_clean":"2024-09-16T13:30:08.994101+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:08.994101+0000","last_undegraded":"2024-09-16T13:30:08.994101+0000","last_fullsized":"2024-09-16T13:30:08.994101+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054439960000000004,"stat_sum":{"num_bytes":39641212,"num_objects":45,"num_object_clones":6,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":20,"num_read":6213,"num_read_kb":25005,"num_write":512,"num_write_kb":8615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.d","version":"120'392","reported_seq":2955,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:08.002364+0000","last_change":"2024-09-16T13:30:08.002364+0000","last_active":"2024-09-16T13:30:08.002364+0000","last_peered":"2024-09-16T13:30:08.002364+0000","last_clean":"2024-09-16T13:30:08.002364+0000","last_became_active":"2024-09-16T13:29:55.469723+0000","last_became_peered":"2024-09-16T13:29:55.469723+0000","last_unstale":"2024-09-16T13:30:08.002364+0000","last_undegraded":"2024-09-16T13:30:08.002364+0000","last_fullsized":"2024-09-16T13:30:08.002364+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.171374+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0050863289999999997,"stat_sum":{"num_bytes":28818625,"num_objects":53,"num_object_clones":12,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":22,"num_read":2888,"num_read_kb":22083,"num_write":419,"num_write_kb":9806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"a"}]},{"pgid":"4.e","version":"121'341","reported_seq":1633,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T13:30:09.042409+0000","last_change":"2024-09-16T13:30:08.999050+0000","last_active":"2024-09-16T13:30:09.042409+0000","last_peered":"2024-09-16T13:30:09.042409+0000","last_clean":"2024-09-16T13:30:09.042409+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:09.042409+0000","last_undegraded":"2024-09-16T13:30:09.042409+0000","last_fullsized":"2024-09-16T13:30:09.042409+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010242243999999999,"stat_sum":{"num_bytes":16623415,"num_objects":41,"num_object_clones":7,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":24,"num_read":1710,"num_read_kb":17531,"num_write":404,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1c","length":"b"}]},{"pgid":"4.f","version":"119'798","reported_seq":3237,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T13:30:07.956276+0000","last_change":"2024-09-16T13:30:07.956471+0000","last_active":"2024-09-16T13:30:07.956276+0000","last_peered":"2024-09-16T13:30:07.956276+0000","last_clean":"2024-09-16T13:30:07.956276+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:07.956276+0000","last_undegraded":"2024-09-16T13:30:07.956276+0000","last_fullsized":"2024-09-16T13:30:07.956276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050117399999999996,"stat_sum":{"num_bytes":26695036,"num_objects":48,"num_object_clones":15,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":24,"num_read":5460,"num_read_kb":14685,"num_write":1102,"num_write_kb":7902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"a"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1051310255,"num_objects":1373,"num_object_clones":304,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":553,"num_read":158923,"num_read_kb":798095,"num_write":20313,"num_write_kb":460605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45686784,"data_stored":452262261,"data_compressed":10563680,"data_compressed_allocated":36225024,"data_compressed_original":446155667,"omap_allocated":0,"internal_metadata":0},"log_size":16370,"ondisk_log_size":16370,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":168,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509469,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306680,"kb_used_data":5088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94065160,"statfs":{"total":96636764160,"available":96322723840,"internally_reserved":0,"allocated":5210112,"data_stored":35440733,"data_compressed":682898,"data_compressed_allocated":2945024,"data_compressed_original":34201600,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.504},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.29699999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.32200000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.34799999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509471,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309900,"kb_used_data":8368,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94061940,"statfs":{"total":96636764160,"available":96319426560,"internally_reserved":0,"allocated":8568832,"data_stored":74483524,"data_compressed":1538949,"data_compressed_allocated":5599232,"data_compressed_original":72766088,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.374}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.35999999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.439},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.44600000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509470,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304828,"kb_used_data":7444,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94067012,"statfs":{"total":96636764160,"available":96324620288,"internally_reserved":0,"allocated":7622656,"data_stored":51777699,"data_compressed":1808643,"data_compressed_allocated":4882432,"data_compressed_original":50167143,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.442},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.34399999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.315},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.24399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.35699999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41999999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509469,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312968,"kb_used_data":7160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94058872,"statfs":{"total":96636764160,"available":96316284928,"internally_reserved":0,"allocated":7331840,"data_stored":43743928,"data_compressed":1492042,"data_compressed_allocated":4259840,"data_compressed_original":41998019,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.20000000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.32300000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41399999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.45100000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542175,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":316028,"kb_used_data":6188,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94055812,"statfs":{"total":96636764160,"available":96313151488,"internally_reserved":0,"allocated":6336512,"data_stored":57154839,"data_compressed":1128376,"data_compressed_allocated":4100096,"data_compressed_original":55953060,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.373},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.45400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.33000000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.318},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.35399999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309268,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062572,"statfs":{"total":96636764160,"available":96320073728,"internally_reserved":0,"allocated":3674112,"data_stored":18142741,"data_compressed":661055,"data_compressed_allocated":1646592,"data_compressed_original":17018880,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.316},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53200000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317216,"kb_used_data":11528,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94054624,"statfs":{"total":96636764160,"available":96311934976,"internally_reserved":0,"allocated":11804672,"data_stored":103604392,"data_compressed":2471193,"data_compressed_allocated":8347648,"data_compressed_original":101568372,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.247}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.625},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.65000000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305216,"kb_used_data":7856,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94066624,"statfs":{"total":96636764160,"available":96324222976,"internally_reserved":0,"allocated":8044544,"data_stored":80073711,"data_compressed":1280989,"data_compressed_allocated":5545984,"data_compressed_original":78646845,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.316}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38100000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.372},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.44800000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.39100000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.44},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6557696,"data_stored":78769112,"data_compressed":1275390,"data_compressed_allocated":5509120,"data_compressed_original":78073405,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9756672,"data_stored":99109041,"data_compressed":2333449,"data_compressed_allocated":7811072,"data_compressed_original":97853440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2076672,"data_stored":16697177,"data_compressed":646131,"data_compressed_allocated":1548288,"data_compressed_original":16363520,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5705728,"data_stored":42738934,"data_compressed":1370259,"data_compressed_allocated":4116480,"data_compressed_original":41744067,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6070272,"data_stored":50914221,"data_compressed":1746418,"data_compressed_allocated":4808704,"data_compressed_original":50052455,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6877184,"data_stored":73463588,"data_compressed":1387189,"data_compressed_allocated":5423104,"data_compressed_original":72503944,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4911104,"data_stored":56440691,"data_compressed":1128376,"data_compressed_allocated":4100096,"data_compressed_original":55953060,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3731456,"data_stored":34129497,"data_compressed":676468,"data_compressed_allocated":2908160,"data_compressed_original":33611776,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:12.931 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:12.922+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:12.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:30:13.025 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:13.023+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:13.088 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.088 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:13.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.088 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:111344 4.6 4.b318b706 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:30:13.086492+0000", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.3624000000000002e-05, 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4441000000000003e-05, 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "queued for pg", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 111344 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086492+0000", 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086492+0000", 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086493+0000", 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4720000000000001e-06 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086499+0000", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086513+0000", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4307999999999999e-05 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:30:13.086520+0000", 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.091 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.092 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:13.092 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 1 2024-09-16T13:30:13.092 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:13.126 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:13.124+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:13.204 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.204 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:13.204 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:13.204 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:13.205 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:13.226 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:13.225+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:13.314 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 96.145651795999996, 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:30:13.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 96.140176230999998, 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:13.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:30:13.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:30:13.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.379849424, 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:30:13.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:30:13.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:30:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.363311299000003, 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.020607523999999, 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:30:13.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:30:13.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.020591719999999, 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.020576159000001, 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.020558445000006, 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:30:13.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.020531861999999, 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:30:13.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 94.178977949, 2024-09-16T13:30:13.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:30:13.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.367 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.981538377999996, 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:30:13.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:30:13.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:13.373 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 73.406661772000007, 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:30:13.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.377 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.059371814000002, 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:30:13.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:30:13.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.059091917999993, 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:30:13.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:30:13.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.627400201999997, 2024-09-16T13:30:13.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:30:13.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:30:13.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:13.397 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.033328820000001, 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:30:13.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:30:13.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.964599118999999, 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:30:13.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:30:13.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.95373438, 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:30:13.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:30:13.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100190 4.1e 4:7a9178f5:::rbd_data.11a97e9c0852.0000000000000012:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.509650586999999, 2024-09-16T13:30:13.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048355217999999998, 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100190 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801667+0000", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801670+0000", 2024-09-16T13:30:13.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801678+0000", 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3119999999999996e-06 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801681+0000", 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801710+0000", 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9496999999999999e-05 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801752+0000", 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1288000000000002e-05 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.804108+0000", 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023560870000000002 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849951+0000", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045842898999999999 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849959+0000", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5180000000000004e-06 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849995+0000", 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6241000000000001e-05 2024-09-16T13:30:13.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.850022+0000", 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6336999999999999e-05 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103827 4.1e 4:7b9e4072:::rbd_data.11a976e61a66.0000000000000020:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.118605155000001, 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046988757999999999, 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103827 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192713+0000", 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9000000000000005e-07 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192718+0000", 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7060000000000003e-06 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192728+0000", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0730999999999999e-05 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192733+0000", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6079999999999998e-06 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192774+0000", 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0859999999999998e-05 2024-09-16T13:30:13.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192861+0000", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7551000000000003e-05 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.195707+0000", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028459779999999999 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239591+0000", 2024-09-16T13:30:13.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043883414000000003 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239603+0000", 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1843e-05 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239660+0000", 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7012000000000002e-05 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239701+0000", 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1065000000000001e-05 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.425 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:13.427 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:13.424+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:13.525 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.525 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:13.526 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:13.526 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.527 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:13.525+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:13.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:13.628 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:13.626+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:13.672 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.672 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:13.672 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:13.672 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:13.672 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:13.765 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:13.763+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:13.821 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:13.821 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.116293252999995, 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:30:13.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.095229345000007, 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:30:13.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.188259084999999, 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.832 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:13.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:30:13.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032325980000003, 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.838 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:30:13.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:30:13.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032310111000001, 2024-09-16T13:30:13.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:30:13.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:30:13.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032295073999997, 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.847 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:13.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032280407000002, 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:30:13.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032265666000001, 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:30:13.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032247847000001, 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:30:13.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:30:13.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:30:13.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032063440000002, 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:30:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:30:13.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032045343999997, 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:30:13.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.875 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032024497999998, 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:30:13.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.032009703, 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.880 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.884 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.031995096000003, 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:30:13.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.03198064, 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.889 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:30:13.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:30:13.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:13.893 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.536090203999997, 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:30:13.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:30:13.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.086820193999998, 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.898 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:30:13.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:30:13.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.444678387, 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:30:13.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:30:13.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:30:13.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.070247754, 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:30:13.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:30:13.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:30:13.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.058845863999998, 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:13.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:13.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:13.920 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:13.921 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:13.922 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:13.919+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:14.000 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:30:14.022 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:14.020+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:14.078 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.079 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:14.079 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:14.079 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:14.132 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:14.130+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:14.189 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.189 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:14.189 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:14.189 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:14.189 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:14.224 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:14.222+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:14.325 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:14.324+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:14.337 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.337 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:14.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 100.20302988900001, 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:30:14.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.812211974999997, 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:30:14.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:30:14.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.174537505999993, 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:30:14.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:30:14.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.019505468999995, 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:14.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:14.361 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.014210191999993, 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:14.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:30:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.013651633999999, 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:30:14.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:14.373 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.340998130999999, 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:14.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:30:14.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.340971885999998, 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:30:14.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.340951978, 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.383 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.764473975000001, 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:30:14.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:30:14.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:30:14.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.599126337000001, 2024-09-16T13:30:14.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:30:14.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.582933592000003, 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:30:14.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.456215690999997, 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.404 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:30:14.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:30:14.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:14.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.382385552000002, 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:30:14.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:30:14.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.263644315999997, 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:30:14.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 40.251472626999998, 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:14.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:30:14.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.696714128, 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:30:14.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:30:14.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.851935469000001, 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:30:14.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:30:14.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.842139060000001, 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:30:14.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 23.899238852, 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.438 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:30:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:30:14.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:30:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.444 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:14.446 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:14.444+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:14.547 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:14.545+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:14.584 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.584 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:14.584 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:14.584 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:14.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:14.647 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:14.646+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:14.693 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.693 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:14.693 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:14.693 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:14.693 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:14.748 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:14.746+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:14.764 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:14.765 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:14.780 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:14.795 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:14.795 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:14.806 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560951553999999, 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:14.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:14.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:14.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.812 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560889080999999, 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:14.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:14.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560844427999996, 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:14.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:14.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560771326999998, 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:14.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:14.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:14.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:14.830 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560752820999994, 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:14.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:14.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560578868999997, 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:14.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:14.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.560540314999997, 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:14.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:14.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.848 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.559581621000007, 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:14.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:14.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.559523823999996, 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:14.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:14.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.559429948000002, 2024-09-16T13:30:14.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:14.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.559401958999999, 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:14.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:14.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:14.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.558127932999994, 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:14.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.558104686999997, 2024-09-16T13:30:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:14.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:14.885 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.830758537999998, 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:14.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:14.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.526130635999998, 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.892 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:14.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:14.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.513409805999999, 2024-09-16T13:30:14.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:14.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.290376582, 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:14.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.810803503999999, 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:14.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:14.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.933087127, 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:14.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 28.915952827000002, 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:14.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:14.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:14.927 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:14.928 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:14.929 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:14.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:14.930 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:14.931 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:14.931 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:14.932 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:14.933 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:14.927+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:15.029 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:15.028+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:15.046 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.046 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:15.131 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:15.129+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:15.194 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.195 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:15.231 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:15.230+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:15.301 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.301 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:15.358 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:15.356+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:15.412 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.412 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:15.433 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:15.432+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:15.534 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:15.532+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:15.561 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.561 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:15.635 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:15.634+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:15.667 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.668 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:15.736 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:15.734+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:15.774 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.774 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:15.866 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:15.865+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:15.921 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:15.921 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:15.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:15.971 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:15.969+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:16.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:30:16.067 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:16.066+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:16.069 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.069 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:16.140 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:16.138+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:16.188 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.188 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:16.241 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:16.239+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:16.293 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.294 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:16.343 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:16.341+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:16.440 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.440 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.443 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:16.441+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:16.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:16.543 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:16.542+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:16.547 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.547 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:16.644 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:16.643+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:16.654 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.654 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:16.746 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:16.744+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:16.800 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.800 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.808 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:16.847 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:16.845+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:16.908 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:16.908 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:16.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:16.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:16.961 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:16.959+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:17.006 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:30:17.024 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:17.024 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:17.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:17.075 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:17.073+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:17.087 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:17.087 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:17.106 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:17.106 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:17.137 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:17.137 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:17.149 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:17.147+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:17.165 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:17.166 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:17.181 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:17.181 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:17.197 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:17.197 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 98.494801245999994, 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:30:17.200 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:30:17.205 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 98.473737338000007, 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:17.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:30:17.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.209 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.566767077999998, 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.210 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:30:17.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:30:17.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:30:17.215 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410833973000003, 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.216 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:30:17.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410818104000001, 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.221 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:30:17.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.225 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410803067000003, 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:17.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:30:17.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410788400000001, 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:30:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:30:17.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410773659, 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:30:17.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:17.239 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.41075584, 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:30:17.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410571433000001, 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.244 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:30:17.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:17.248 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410553337000003, 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:30:17.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.252 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410532490999998, 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.253 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:30:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410517695999999, 2024-09-16T13:30:17.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:30:17.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410503089000002, 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.410488633, 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:30:17.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:30:17.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 49.914598196999997, 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:30:17.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:30:17.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:30:17.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.465328186999997, 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:30:17.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:17.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.281 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.823186379999999, 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:30:17.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.448755747, 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:30:17.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:30:17.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:30:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:30:17.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.437353857000002, 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:30:17.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:30:17.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:30:17.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.299 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:17.300 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:17.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:17.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:17.302 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:17.299+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:17.401 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:17.399+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:17.440 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:17.440 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:17.440 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:17.440 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:17.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:17.501 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:17.500+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:17.587 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:17.587 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:17.587 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:17.587 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:17.587 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:17.595 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:17.602 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:17.600+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 103.559354584, 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:30:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:30:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:30:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:30:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 73.168536669999995, 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:30:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:30:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.530862201000005, 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:30:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:30:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:30:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.375830164000007, 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:30:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.370534887000005, 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:30:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:30:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 72.369976328999996, 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:30:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:30:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 66.697322826000004, 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:30:17.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 66.697296581000003, 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.736 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:30:17.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 66.697276673000005, 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.120798669999999, 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:30:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.955451031999999, 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:30:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:30:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.939258287000001, 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:30:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:30:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.812540386000002, 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:30:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:30:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:30:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.738710247, 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.619969011000002, 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:30:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:30:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.607797322000003, 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:30:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:30:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.053038823000001, 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:30:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.208260163999999, 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:30:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.198463754999999, 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:30:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.255563547000001, 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:30:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:30:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:30:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:17.802 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:17.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:17.804 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:17.802+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:17.904 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:17.904 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:17.905 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:17.903+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:17.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:17.933 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:30:17.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T13:30:18.005 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:18.003+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:18.013 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:18.014 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:18.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:18.106 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:18.104+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.137 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891881452000007, 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:18.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:18.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891818978999993, 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.143 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:18.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:18.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:18.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.148 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891774326000004, 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.149 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891701225000006, 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891682719000002, 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.160 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.165 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891508767000005, 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:18.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:18.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.891470213000005, 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:18.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:18.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:18.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.890511519, 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:18.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:18.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:18.183 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.890453722000004, 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.890359845999996, 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:18.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:18.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:18.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.890331857000007, 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.195 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:18.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:18.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.889057831000002, 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.201 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.889034585000005, 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.161688436000006, 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.857060533999999, 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.844339703999999, 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:18.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.621306480000001, 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:18.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:18.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 50.141733402, 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:18.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:18.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:18.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:18.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.264017025000001, 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:18.240 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:18.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.245 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.246882724999999, 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:18.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:18.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:18.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:18.250 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:18.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:18.252 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:18.249+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:18.351 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:18.351 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:18.351 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:18.352 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:18.350+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:18.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:18.452 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:18.450+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:18.498 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:18.498 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:18.498 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:18.498 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:18.498 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:18.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:18.592 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:18.589+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 100.934628679, 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.648 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:30:18.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:30:18.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:30:18.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 100.925089568, 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:30:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:30:18.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 91.958632832999996, 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:30:18.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:30:18.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.248221381999997, 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.665 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:30:18.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:30:18.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.671 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 87.237574332999998, 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:18.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:30:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 78.570512751999999, 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:30:18.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:30:18.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.002881696000003, 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.682 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:30:18.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:30:18.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.687 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 53.985748802000003, 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.025359856000001, 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.692 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:30:18.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.008871135, 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:30:18.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:30:18.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.628986097999999, 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.702 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:30:18.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:30:18.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.628393168999999, 2024-09-16T13:30:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:30:18.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.628378505999997, 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:30:18.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.333801854000001, 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.715 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:30:18.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:30:18.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:18.721 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.320520627999997, 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:30:18.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:30:18.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:18.725 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.355270953000002, 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:30:18.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:30:18.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.338994223999997, 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:30:18.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.06764931, 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.736 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:30:18.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:30:18.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.067454415, 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:30:18.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:30:18.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 32.048191496999998, 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:30:18.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:30:18.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:18.751 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:18.752 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:18.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:18.753 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:18.751+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:18.854 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:18.852+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:18.895 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:18.895 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:18.895 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:18.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:18.987 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:18.986+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:19.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:30:19.030 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T13:30:19.046 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:19.046 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:19.046 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:19.046 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:19.046 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:19.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T13:30:19.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:19.089 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:19.087+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:19.157 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:19.155+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:19.204 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:19.204 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.47196732899999, 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.205 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.206 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:30:19.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.208 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:30:19.209 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:30:19.210 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 102.029076007, 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.213 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.214 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.216 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.515617978999998, 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:30:19.219 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:30:19.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:19.222 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.505528712, 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:30:19.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:30:19.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 69.226815685000005, 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.227 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 64.278656712, 2024-09-16T13:30:19.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:30:19.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:30:19.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:30:19.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.240 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191476727000001, 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:19.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.244 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191459129000002, 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:30:19.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:30:19.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191444429999997, 2024-09-16T13:30:19.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191433498999999, 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:30:19.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:30:19.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191417880000003, 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:30:19.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:19.262 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191396617000002, 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191381606999997, 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.267 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:30:19.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.271 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191365525999998, 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:19.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191325659, 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:30:19.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:30:19.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.191314306999999, 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.280 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.466763421000003, 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:30:19.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.465475095000002, 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.288 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:30:19.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:30:19.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:30:19.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.465203160000002, 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:30:19.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:19.298 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.459724538000003, 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:30:19.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:30:19.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:19.302 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:19.303 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:19.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:19.304 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:19.302+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:19.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:19.404 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:19.402+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:19.417 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:19.418 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:19.433 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:19.443 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:19.443 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:19.444 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:19.444 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:19.447 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:19.448 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:19.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:19.462 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:19.462 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:19.478 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:19.478 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:19.493 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:19.494 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:19.509 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:19.509 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:19.524 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:19.524 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:19.535 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:19.534+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:19.589 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:19.710 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:19.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:19.712 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:19.710+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:19.813 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:19.811+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:19.850 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:19.850 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:19.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:19.913 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:19.911+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:19.926 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:30:19.931 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T13:30:19.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T13:30:19.962 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:19.962 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:19.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:20.031 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:20.029+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:20.112 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.113 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.115 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:20.113+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:20.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:20.216 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:20.214+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:20.221 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.221 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.230 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:20.317 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:20.315+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:20.330 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.330 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.338 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:20.418 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:20.416+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:20.476 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.476 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:20.519 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:20.517+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:20.584 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.585 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:20.637 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:20.635+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:20.691 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.692 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:20.720 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:20.719+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:20.821 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:20.819+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:20.838 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.838 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:20.921 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:20.920+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:20.985 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:20.985 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:20.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:21.034 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T13:30:21.034 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:21.032+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:21.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T13:30:21.108 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:21.108 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:21.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:21.159 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:21.157+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:21.225 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:21.223+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:21.262 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:21.262 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:21.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:21.325 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:21.323+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:21.370 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:21.370 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:21.379 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:21.426 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:21.424+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:21.477 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:21.478 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:21.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:21.528 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:21.526+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:21.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:21.626 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:21.626 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:21.628 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:21.626+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:21.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:21.688 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:21.705 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:21.705 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:21.720 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:21.721 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:21.729 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:21.727+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:21.736 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:21.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495921009, 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495858536, 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495813882999997, 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495740781999999, 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495722275999995, 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495548323999998, 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.495509769999998, 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.494551075999993, 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.494493278999997, 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.494399403000003, 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.494371414, 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.493097387999995, 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.493074141999998, 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.765727992999999, 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.461100090999999, 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.448379260999999, 2024-09-16T13:30:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.225346037000001, 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 53.745772959, 2024-09-16T13:30:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.868056582000001, 2024-09-16T13:30:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.850922281999999, 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:21.878 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:21.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:21.880 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:21.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:21.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:21.882 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:21.878+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:21.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:30:21.991 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:21.989+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:21.993 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:21.993 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:21.993 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:21.993 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.001 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:22.001 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:22.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:22.048 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T13:30:22.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T13:30:22.105 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:22.103+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:22.108 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.108 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:22.108 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:22.108 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:22.109 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:22.181 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:22.179+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.506668893, 2024-09-16T13:30:22.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:30:22.222 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:30:22.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.224 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:30:22.225 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:30:22.226 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.497129782, 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:22.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 95.530673046999993, 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.231 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:22.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:30:22.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:30:22.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.820261595999995, 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:30:22.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:30:22.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.244 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 90.809614546999995, 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:22.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:30:22.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:30:22.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:30:22.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.250 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.250 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:30:22.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:30:22.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:30:22.251 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 82.142552965999997, 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.252 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:30:22.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:30:22.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:30:22.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:30:22.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:30:22.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:30:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:30:22.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:30:22.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:22.260 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 57.57492191, 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:30:22.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 57.557789016000001, 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:30:22.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.597400069999999, 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:30:22.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:30:22.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:30:22.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:30:22.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:30:22.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:30:22.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.282 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.580911348999997, 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:30:22.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:30:22.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.201026312000003, 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.287 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.291 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.200433382999996, 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:30:22.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.200418720000002, 2024-09-16T13:30:22.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:30:22.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:30:22.300 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.905842067999998, 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.301 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 44.892560842000002, 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:30:22.307 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:22.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:30:22.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:30:22.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:22.312 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.927311166999999, 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:22.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:22.319 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.911034438000002, 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:30:22.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.323 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.639689523999998, 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:30:22.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.329 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.639494628999998, 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:30:22.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:30:22.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:30:22.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.620231711000002, 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:30:22.335 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.339 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:22.341 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:22.339+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:22.440 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:22.440 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:22.440 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.441 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:22.440+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:22.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:22.542 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:22.541+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:22.547 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.547 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:22.547 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:22.547 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:22.547 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:22.643 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:22.642+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:22.697 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.697 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:22.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:22.697 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:22.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.698 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:22.698 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:22.698 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 107.964023633, 2024-09-16T13:30:22.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:30:22.698 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:30:22.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:30:22.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:30:22.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:30:22.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:30:22.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:30:22.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:30:22.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:30:22.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:30:22.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:30:22.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:30:22.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:30:22.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:30:22.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:30:22.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:30:22.711 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.712 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.712 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.712 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:30:22.713 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:22.713 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.521132311, 2024-09-16T13:30:22.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:30:22.713 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.713 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.714 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:22.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:30:22.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:30:22.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:30:22.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:30:22.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:30:22.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:30:22.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:30:22.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:30:22.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:30:22.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:30:22.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.723 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 96.007674283, 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:30:22.724 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.725 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.725 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.725 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.725 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.725 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:22.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:30:22.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:30:22.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:30:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:30:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:30:22.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:30:22.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:30:22.737 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.737 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.737 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 95.997585016000002, 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.738 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:30:22.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:30:22.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:30:22.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:30:22.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:30:22.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:30:22.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:30:22.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:30:22.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:22.747 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:22.748 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 72.718871988999993, 2024-09-16T13:30:22.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:30:22.748 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.748 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.748 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:22.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:30:22.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:30:22.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:30:22.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:30:22.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:30:22.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:30:22.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:30:22.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:30:22.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.757 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:30:22.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:30:22.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:30:22.760 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 67.770713016000002, 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.761 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:30:22.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:30:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683533031000003, 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.767 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:30:22.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.771 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683515432999997, 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:30:22.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:30:22.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:30:22.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683500733999999, 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:30:22.776 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.780 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683489803000001, 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:30:22.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683474183999998, 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683452920999997, 2024-09-16T13:30:22.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:22.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:30:22.793 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683437910999999, 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.794 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.796 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:30:22.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.68342183, 2024-09-16T13:30:22.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:30:22.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:30:22.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683381963000002, 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:30:22.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:22.807 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 48.683370611000001, 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:22.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:30:22.811 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.958819724999998, 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:30:22.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:30:22.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.816 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.957531398999997, 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:30:22.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:22.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.957259464000003, 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:30:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 40.951780841999998, 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:22.827 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:22.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:30:22.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:22.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:22.832 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:22.834 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:22.831+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:22.934 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:22.932+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:22.972 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:22.972 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:22.972 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:22.972 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:22.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:23.050 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:30:23.054 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T13:30:23.064 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:23.063+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:23.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T13:30:23.125 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:23.125 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:23.125 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:23.125 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:23.125 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:23.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:23.175 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:23.173+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:23.236 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:23.234+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:23.273 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:1548 4.a 4:55819aa3:::rbd_data.11a962dad130.00000000000000a5:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.58409355800001, 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051877126000000003, 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.274 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1548 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687019+0000", 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687021+0000", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4649999999999999e-06 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687026+0000", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687040+0000", 2024-09-16T13:30:23.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4098e-05 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687050+0000", 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8369999999999992e-06 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687085+0000", 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4541e-05 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.687133+0000", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8909999999999998e-05 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.689283+0000", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021493440000000001 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738810+0000", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049526769999999998 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738819+0000", 2024-09-16T13:30:23.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1989999999999995e-06 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738857+0000", 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8701000000000001e-05 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738896+0000", 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8711000000000003e-05 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:1548 4.a e37/32 ondisk, result = 0)", 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.57452207999999, 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042221538000000003, 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696591+0000", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696592+0000", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8200000000000008e-07 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696595+0000", 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2799999999999999e-06 2024-09-16T13:30:23.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696607+0000", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1798e-05 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.696630+0000", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738795+0000", 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042165828000000002 2024-09-16T13:30:23.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738803+0000", 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9889999999999992e-06 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:33.738812+0000", 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7199999999999995e-06 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7498 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 14614528~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.727093139, 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038008976999999999, 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7498 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544019+0000", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544020+0000", 2024-09-16T13:30:23.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.11e-07 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544021+0000", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0079999999999999e-06 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544024+0000", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0259999999999999e-06 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.544026+0000", 2024-09-16T13:30:23.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.875e-06 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581724+0000", 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037698201000000001 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.581775+0000", 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1455000000000001e-05 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.582028+0000", 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025320099999999999 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8166 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 1769472~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980825979, 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037856426999999998, 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.288 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8166 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290287+0000", 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.03e-07 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290288+0000", 2024-09-16T13:30:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2900000000000002e-07 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290291+0000", 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8820000000000001e-06 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290293+0000", 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8530000000000001e-06 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327850+0000", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037557950999999999 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.327902+0000", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328143+0000", 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000240585 2024-09-16T13:30:23.292 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8167 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 2686976~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980818238, 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038381480000000003, 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.293 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8167 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290294+0000", 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0800000000000001e-07 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290295+0000", 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7899999999999997e-07 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290299+0000", 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290301+0000", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.686e-06 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328178+0000", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037877297999999997 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328267+0000", 2024-09-16T13:30:23.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9137999999999999e-05 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328676+0000", 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040845600000000002 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.297 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8168 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980809679, 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038667908000000001, 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8168 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.298 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290303+0000", 2024-09-16T13:30:23.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0699999999999999e-07 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290304+0000", 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7000000000000003e-07 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290307+0000", 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7640000000000001e-06 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290309+0000", 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7460000000000001e-06 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328689+0000", 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038380818999999997 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328733+0000", 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3421999999999999e-05 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.328971+0000", 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023808 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8169 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980800802, 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039117135999999997, 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.302 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8169 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290312+0000", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290317+0000", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4500000000000003e-06 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290322+0000", 2024-09-16T13:30:23.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290324+0000", 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329007+0000", 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038683429999999998 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329096+0000", 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9248999999999999e-05 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329429+0000", 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033260199999999998 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.308 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8170 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980786937, 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039460856000000002, 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8170 2024-09-16T13:30:23.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290326+0000", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000001e-07 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290327+0000", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290330+0000", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5920000000000001e-06 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290332+0000", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.86e-06 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329443+0000", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039110565999999999 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329487+0000", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4551000000000001e-05 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329787+0000", 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029910899999999999 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8171 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.98077847, 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039884615999999998, 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.313 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8171 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:23.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290334+0000", 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290335+0000", 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2200000000000002e-07 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290339+0000", 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5080000000000001e-06 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290341+0000", 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3369999999999998e-06 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329822+0000", 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039481186000000001 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.329914+0000", 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1335999999999994e-05 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330219+0000", 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030511699999999999 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.317 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:8172 4.a 4:50a6fbe1:::rbd_data.11a9df936547.0000000000000001:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 104.980769667, 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040227349000000003, 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 8172 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290343+0000", 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.05e-07 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290344+0000", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5000000000000001e-07 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290348+0000", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.749e-06 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.290350+0000", 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8819999999999999e-06 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330263+0000", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039913218 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330313+0000", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:38.330570+0000", 2024-09-16T13:30:23.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000257722 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28043 4.5 4:a1fa080e:::rbd_data.11a9320547a2.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.357299608999995, 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077818496000000001, 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.322 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28043 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913813+0000", 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913814+0000", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913818+0000", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.912e-06 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913829+0000", 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1591e-05 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913834+0000", 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5709999999999999e-06 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913880+0000", 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5581000000000003e-05 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.913929+0000", 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9616000000000001e-05 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.917179+0000", 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032501539999999999 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991504+0000", 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.074324764000000001 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991573+0000", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9287000000000003e-05 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991604+0000", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0979000000000003e-05 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991632+0000", 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-09-16T13:30:23.327 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28043 4.5 e51/32 ondisk, result = 0)", 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.344070927999994, 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064465529999999993, 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927042+0000", 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7199999999999998e-07 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927046+0000", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3469999999999999e-06 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927057+0000", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1693e-05 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.927064+0000", 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1410000000000002e-06 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991444+0000", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.064380003000000005 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991450+0000", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7150000000000001e-06 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:50.991507+0000", 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6959000000000001e-05 2024-09-16T13:30:23.331 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38887 4.a 4:54988710:::rbd_data.11a9a07602f8.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 81.830244738999994, 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090299272999999999, 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38887 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440868+0000", 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.440878+0000", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7550000000000006e-06 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442210+0000", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013321030000000001 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442223+0000", 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3271999999999999e-05 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442230+0000", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9580000000000001e-06 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442275+0000", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4878e-05 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.442375+0000", 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010045800000000001 2024-09-16T13:30:23.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.446499+0000", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041240319999999997 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531083+0000", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.084584096999999997 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531094+0000", 2024-09-16T13:30:23.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0781000000000001e-05 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531136+0000", 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2067999999999998e-05 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531167+0000", 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0871000000000003e-05 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.337 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:38887 4.a e61/32 ondisk, result = 0)", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 81.813188861, 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073163878000000002, 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457924+0000", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457925+0000", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3340000000000001e-06 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457930+0000", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6859999999999997e-06 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457960+0000", 2024-09-16T13:30:23.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0349e-05 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.457984+0000", 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4309999999999999e-05 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531038+0000", 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073053811999999996 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531050+0000", 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1368e-05 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:01.531088+0000", 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8019000000000003e-05 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75189 4.5 4:a70dbba6:::rbd_data.11a9cafed80f.0000000000000054:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:23.341 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.909038281000001, 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063728125999999996, 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75189 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.342 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362074+0000", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362077+0000", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6170000000000001e-06 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362087+0000", 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8630000000000003e-06 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362121+0000", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4579000000000002e-05 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362134+0000", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2393e-05 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362275+0000", 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141304 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.344 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.362410+0000", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000135217 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.365362+0000", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0029521740000000001 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425698+0000", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.060335105 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.345 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425726+0000", 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8229999999999999e-05 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425761+0000", 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4730000000000001e-05 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425802+0000", 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1913999999999999e-05 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.346 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75189 4.5 e80/32 ondisk, result = 0)", 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.898399241, 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.053093882000000002, 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372713+0000", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372714+0000", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372717+0000", 2024-09-16T13:30:23.348 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8420000000000001e-06 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372727+0000", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.372732+0000", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0139999999999998e-06 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425716+0000", 2024-09-16T13:30:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052983817000000002 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425722+0000", 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8980000000000001e-06 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.425807+0000", 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5342999999999999e-05 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.350 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100149 4.1f 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.538926089, 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037897414999999997, 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100149 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.351 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732187+0000", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.732188+0000", 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8050000000000001e-06 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.352 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734548+0000", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002359758 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734560+0000", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1919999999999999e-05 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734569+0000", 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0620000000000007e-06 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.353 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734606+0000", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6939e-05 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734726+0000", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119768 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735030+0000", 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000304109 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.354 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765403+0000", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030373052000000001 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.769969+0000", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045659089999999999 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770038+0000", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8748e-05 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.355 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770084+0000", 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6344999999999998e-05 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100148 4.12 e95/32 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head v 95'622, mlcod=92'620)", 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.536392821, 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037682625999999997, 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.356 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.734720+0000", 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:30:23.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737443+0000", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027229519999999998 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737460+0000", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6569e-05 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737471+0000", 2024-09-16T13:30:23.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1401e-05 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737528+0000", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7046999999999998e-05 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.737551+0000", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2459e-05 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772360+0000", 2024-09-16T13:30:23.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034809253999999998 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772402+0000", 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2459999999999997e-05 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100332 4.1f 4:f86ca47e:::rbd_data.11a97e9c0852.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.245711319999998, 2024-09-16T13:30:23.360 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054467106000000001, 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100332 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.361 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025401+0000", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025403+0000", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6330000000000001e-06 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025408+0000", 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9049999999999996e-06 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.362 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025420+0000", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2029e-05 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025424+0000", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4819999999999996e-06 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025459+0000", 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4489000000000001e-05 2024-09-16T13:30:23.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.025514+0000", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4737000000000001e-05 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.028379+0000", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028655379999999999 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079727+0000", 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051347589999999999 2024-09-16T13:30:23.364 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079800+0000", 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3681000000000001e-05 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079840+0000", 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9243999999999999e-05 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079868+0000", 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:100332 4.1f e95/32 ondisk, result = 0)", 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 41.234943671000003, 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043713039000000002, 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.366 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036169+0000", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036170+0000", 2024-09-16T13:30:23.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4399999999999999e-07 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036173+0000", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036182+0000", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3689999999999996e-06 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.036187+0000", 2024-09-16T13:30:23.368 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2900000000000002e-06 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079788+0000", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043600379000000002 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079796+0000", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9270000000000005e-06 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:42.079882+0000", 2024-09-16T13:30:23.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6323999999999995e-05 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.370 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:23.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:23.372 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:23.368+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:23.471 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:23.470+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:23.512 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:23.512 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:23.512 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:23.512 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:23.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:23.572 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:23.569+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:23.658 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:23.658 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:23.658 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:23.658 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:23.658 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:23.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:23.673 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:23.670+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 138.47390813800001, 2024-09-16T13:30:23.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:23.772 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.771989203, 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:30:23.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:30:23.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.77196669400001, 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:30:23.778 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:30:23.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.77192036700001, 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.784 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.786 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:30:23.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.788 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.77175360999999, 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.790 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.791 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.792 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.793 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.794 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.795 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.77173884499999, 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.796 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.797 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.798 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.799 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:30:23.801 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.426416528, 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.802 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.803 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.804 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:30:23.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:30:23.806 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.807 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.42637884999999, 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:30:23.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:30:23.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:30:23.811 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.812 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:23.813 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.42635603700001, 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:30:23.815 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.816 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.817 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 137.426270824, 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.819 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.820 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:30:23.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:30:23.823 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.824 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.119691899, 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.825 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.826 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:30:23.828 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.829 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.093783665, 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:30:23.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:30:23.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 109.02501851, 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.834 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:30:23.835 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.836 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:30:23.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 74.523475692999995, 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.839 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.840 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:30:23.842 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 60.328550374000002, 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:30:23.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.845 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.846 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.172673338999999, 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:23.847 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.848 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 49.893721509000002, 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.852 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:30:23.855 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.118366801999997, 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.856 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.857 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.021096381, 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.860 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.862 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:30:23.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 42.019415127999999, 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.865 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.867 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:30:23.868 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.869 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:23.869 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:23.869 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:23.870 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:23.868+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:23.870 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:23.971 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:23.968+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:23.974 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:23.974 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:23.974 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:23.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:24.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T13:30:24.072 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:24.070+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:24.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T13:30:24.142 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:24.142 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:24.142 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:24.142 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:24.142 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:24.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:24.157 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:24.158 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:24.178 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:24.178 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:24.178 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:24.175+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:24.262 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.262 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:24.273 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:24.272+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:24.374 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:24.372+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:24.408 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:24.423 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.423 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.425 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:24.425 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:24.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:24.441 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:24.441 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:24.457 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:24.472 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:24.475 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:24.474+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:24.487 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:24.488 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:24.541 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.541 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:24.577 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:24.575+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:24.647 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.647 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:24.677 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:24.676+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:24.753 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.753 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:24.805 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:24.803+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:24.860 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:24.860 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:24.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:24.879 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:24.877+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:24.928 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T13:30:24.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T13:30:24.989 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:24.986+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:25.028 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.028 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:25.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:30:25.080 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:25.078+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:25.181 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:25.179+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:25.184 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.184 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:25.282 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:25.280+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:25.295 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.295 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:25.383 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:25.382+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:25.442 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.442 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:25.484 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:25.481+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:25.554 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.555 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:25.585 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:25.583+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:25.660 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.660 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:25.713 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:25.711+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:25.767 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.767 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:25.787 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:25.784+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:25.888 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:25.886+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:25.914 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:25.914 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:25.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:25.989 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:25.987+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:26.057 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T13:30:26.057 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2024-09-16T13:30:26.062 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.062 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:26.074 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T13:30:26.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:26.116 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:26.113+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:26.183 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.183 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.191 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:26.188+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:26.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:26.292 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:26.291+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:26.293 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.293 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:26.393 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:26.392+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:26.396 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:26.397 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:26.408 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T13:30:26.052502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384956414,"num_objects":794,"num_object_clones":158,"num_object_copies":1588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":434,"num_read":163276,"num_read_kb":801812,"num_write":21253,"num_write_kb":462427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":13,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17638,"ondisk_log_size":17638,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449520,"kb_used_data":37332,"kb_used_omap":13,"kb_used_meta":2412018,"kb_avail":752525200,"statfs":{"total":773094113280,"available":770585804800,"internally_reserved":0,"allocated":38227968,"data_stored":300256339,"data_compressed":5099687,"data_compressed_allocated":21086208,"data_compressed_original":291456354,"omap_allocated":13632,"internal_metadata":2469907136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-272074323,"num_objects":-126,"num_object_clones":-35,"num_object_copies":-252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-126,"num_whiteouts":-23,"num_read":1369,"num_read_kb":1166,"num_write":212,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009943"},"pg_stats":[{"pgid":"4.1f","version":"135'338","reported_seq":2102,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.161651+0000","last_change":"2024-09-16T13:30:23.109144+0000","last_active":"2024-09-16T13:30:23.161651+0000","last_peered":"2024-09-16T13:30:23.161651+0000","last_clean":"2024-09-16T13:30:23.161651+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:23.161651+0000","last_undegraded":"2024-09-16T13:30:23.161651+0000","last_fullsized":"2024-09-16T13:30:23.161651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032560199999999998,"stat_sum":{"num_bytes":5027510,"num_objects":23,"num_object_clones":2,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":2710,"num_read_kb":20014,"num_write":336,"num_write_kb":9428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1e","version":"135'1071","reported_seq":11739,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.140135+0000","last_change":"2024-09-16T13:30:23.058593+0000","last_active":"2024-09-16T13:30:23.140135+0000","last_peered":"2024-09-16T13:30:23.140135+0000","last_clean":"2024-09-16T13:30:23.140135+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:23.140135+0000","last_undegraded":"2024-09-16T13:30:23.140135+0000","last_fullsized":"2024-09-16T13:30:23.140135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061878769999999996,"stat_sum":{"num_bytes":3083383,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":14,"num_read":14968,"num_read_kb":31050,"num_write":1650,"num_write_kb":9687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1d","version":"136'415","reported_seq":1489,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.064004+0000","last_change":"2024-09-16T13:30:24.064276+0000","last_active":"2024-09-16T13:30:24.064004+0000","last_peered":"2024-09-16T13:30:24.064004+0000","last_clean":"2024-09-16T13:30:24.064004+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:24.064004+0000","last_undegraded":"2024-09-16T13:30:24.064004+0000","last_fullsized":"2024-09-16T13:30:24.064004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0055176310000000003,"stat_sum":{"num_bytes":3000237,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":14,"num_read":1819,"num_read_kb":7961,"num_write":429,"num_write_kb":5828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.1c","version":"135'484","reported_seq":4451,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.093897+0000","last_change":"2024-09-16T13:30:23.074030+0000","last_active":"2024-09-16T13:30:23.093897+0000","last_peered":"2024-09-16T13:30:23.093897+0000","last_clean":"2024-09-16T13:30:23.093897+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:23.093897+0000","last_undegraded":"2024-09-16T13:30:23.093897+0000","last_fullsized":"2024-09-16T13:30:23.093897+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060536599999999998,"stat_sum":{"num_bytes":5225978,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":8,"num_read":7571,"num_read_kb":26873,"num_write":490,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1b","version":"136'468","reported_seq":2303,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.072365+0000","last_change":"2024-09-16T13:30:24.072511+0000","last_active":"2024-09-16T13:30:24.072365+0000","last_peered":"2024-09-16T13:30:24.072365+0000","last_clean":"2024-09-16T13:30:24.072365+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:24.072365+0000","last_undegraded":"2024-09-16T13:30:24.072365+0000","last_fullsized":"2024-09-16T13:30:24.072365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.0071583360000000004,"stat_sum":{"num_bytes":4883732,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":9,"num_read":3240,"num_read_kb":15629,"num_write":533,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.1a","version":"136'398","reported_seq":4170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.155249+0000","last_change":"2024-09-16T13:30:24.067774+0000","last_active":"2024-09-16T13:30:24.155249+0000","last_peered":"2024-09-16T13:30:24.155249+0000","last_clean":"2024-09-16T13:30:24.155249+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:24.155249+0000","last_undegraded":"2024-09-16T13:30:24.155249+0000","last_fullsized":"2024-09-16T13:30:24.155249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0026119400000000001,"stat_sum":{"num_bytes":11030449,"num_objects":23,"num_object_clones":1,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":4026,"num_read_kb":16476,"num_write":437,"num_write_kb":6964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.19","version":"136'386","reported_seq":3428,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.061210+0000","last_change":"2024-09-16T13:30:24.061529+0000","last_active":"2024-09-16T13:30:24.061210+0000","last_peered":"2024-09-16T13:30:24.061210+0000","last_clean":"2024-09-16T13:30:24.061210+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:24.061210+0000","last_undegraded":"2024-09-16T13:30:24.061210+0000","last_fullsized":"2024-09-16T13:30:24.061210+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003079933,"stat_sum":{"num_bytes":26423659,"num_objects":29,"num_object_clones":6,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":14,"num_read":3419,"num_read_kb":15850,"num_write":388,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.18","version":"136'371","reported_seq":3365,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.067186+0000","last_change":"2024-09-16T13:30:24.067410+0000","last_active":"2024-09-16T13:30:24.067186+0000","last_peered":"2024-09-16T13:30:24.067186+0000","last_clean":"2024-09-16T13:30:24.067186+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:24.067186+0000","last_undegraded":"2024-09-16T13:30:24.067186+0000","last_fullsized":"2024-09-16T13:30:24.067186+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.002300182,"stat_sum":{"num_bytes":23217310,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":10,"num_read":3439,"num_read_kb":32799,"num_write":387,"num_write_kb":30744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.17","version":"135'399","reported_seq":3188,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062538+0000","last_change":"2024-09-16T13:30:24.063099+0000","last_active":"2024-09-16T13:30:24.062538+0000","last_peered":"2024-09-16T13:30:24.062538+0000","last_clean":"2024-09-16T13:30:24.062538+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:24.062538+0000","last_undegraded":"2024-09-16T13:30:24.062538+0000","last_fullsized":"2024-09-16T13:30:24.062538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042827439999999998,"stat_sum":{"num_bytes":9490556,"num_objects":28,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":20,"num_read":4459,"num_read_kb":29774,"num_write":362,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.16","version":"135'649","reported_seq":2805,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.064071+0000","last_change":"2024-09-16T13:30:23.064071+0000","last_active":"2024-09-16T13:30:23.064071+0000","last_peered":"2024-09-16T13:30:23.064071+0000","last_clean":"2024-09-16T13:30:23.064071+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:23.064071+0000","last_undegraded":"2024-09-16T13:30:23.064071+0000","last_fullsized":"2024-09-16T13:30:23.064071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011372347999999999,"stat_sum":{"num_bytes":968704,"num_objects":12,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":8,"num_read":4642,"num_read_kb":15931,"num_write":788,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.15","version":"136'808","reported_seq":4060,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062363+0000","last_change":"2024-09-16T13:30:24.062602+0000","last_active":"2024-09-16T13:30:24.062363+0000","last_peered":"2024-09-16T13:30:24.062363+0000","last_clean":"2024-09-16T13:30:24.062363+0000","last_became_active":"2024-09-16T13:29:55.466429+0000","last_became_peered":"2024-09-16T13:29:55.466429+0000","last_unstale":"2024-09-16T13:30:24.062363+0000","last_undegraded":"2024-09-16T13:30:24.062363+0000","last_fullsized":"2024-09-16T13:30:24.062363+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:31.786154+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001939035,"stat_sum":{"num_bytes":50683772,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":15,"num_read":5783,"num_read_kb":34352,"num_write":1210,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.14","version":"132'466","reported_seq":3420,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.061607+0000","last_change":"2024-09-16T13:30:24.061928+0000","last_active":"2024-09-16T13:30:24.061607+0000","last_peered":"2024-09-16T13:30:24.061607+0000","last_clean":"2024-09-16T13:30:24.061607+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:24.061607+0000","last_undegraded":"2024-09-16T13:30:24.061607+0000","last_fullsized":"2024-09-16T13:30:24.061607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031719180000000001,"stat_sum":{"num_bytes":5214933,"num_objects":33,"num_object_clones":9,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":19,"num_read":4915,"num_read_kb":20477,"num_write":499,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.13","version":"136'359","reported_seq":1940,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.068892+0000","last_change":"2024-09-16T13:30:24.069122+0000","last_active":"2024-09-16T13:30:24.068892+0000","last_peered":"2024-09-16T13:30:24.068892+0000","last_clean":"2024-09-16T13:30:24.068892+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:24.068892+0000","last_undegraded":"2024-09-16T13:30:24.068892+0000","last_fullsized":"2024-09-16T13:30:24.068892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0039196500000000002,"stat_sum":{"num_bytes":29790726,"num_objects":28,"num_object_clones":7,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":17,"num_read":1521,"num_read_kb":22420,"num_write":346,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.12","version":"135'794","reported_seq":4142,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.164081+0000","last_change":"2024-09-16T13:30:23.051440+0000","last_active":"2024-09-16T13:30:23.164081+0000","last_peered":"2024-09-16T13:30:23.164081+0000","last_clean":"2024-09-16T13:30:23.164081+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:23.164081+0000","last_undegraded":"2024-09-16T13:30:23.164081+0000","last_fullsized":"2024-09-16T13:30:23.164081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063509500000000002,"stat_sum":{"num_bytes":348190,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":10,"num_read":8606,"num_read_kb":20548,"num_write":964,"num_write_kb":8823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.11","version":"133'748","reported_seq":4756,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.059003+0000","last_change":"2024-09-16T13:30:24.059301+0000","last_active":"2024-09-16T13:30:24.059003+0000","last_peered":"2024-09-16T13:30:24.059003+0000","last_clean":"2024-09-16T13:30:24.059003+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:24.059003+0000","last_undegraded":"2024-09-16T13:30:24.059003+0000","last_fullsized":"2024-09-16T13:30:24.059003+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088673099999999998,"stat_sum":{"num_bytes":3740890,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":15,"num_read":5903,"num_read_kb":16215,"num_write":942,"num_write_kb":25099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.10","version":"135'731","reported_seq":3766,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.058478+0000","last_change":"2024-09-16T13:30:24.058954+0000","last_active":"2024-09-16T13:30:24.058478+0000","last_peered":"2024-09-16T13:30:24.058478+0000","last_clean":"2024-09-16T13:30:24.058478+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:24.058478+0000","last_undegraded":"2024-09-16T13:30:24.058478+0000","last_fullsized":"2024-09-16T13:30:24.058478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058177299999999995,"stat_sum":{"num_bytes":2109440,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":13,"num_read":8256,"num_read_kb":21161,"num_write":987,"num_write_kb":9535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.052852+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:22.052852+0000","last_peered":"2024-09-16T13:30:22.052852+0000","last_clean":"2024-09-16T13:30:22.052852+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:22.052852+0000","last_undegraded":"2024-09-16T13:30:22.052852+0000","last_fullsized":"2024-09-16T13:30:22.052852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"133'487","reported_seq":3543,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.065297+0000","last_change":"2024-09-16T13:30:24.065569+0000","last_active":"2024-09-16T13:30:24.065297+0000","last_peered":"2024-09-16T13:30:24.065297+0000","last_clean":"2024-09-16T13:30:24.065297+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:24.065297+0000","last_undegraded":"2024-09-16T13:30:24.065297+0000","last_fullsized":"2024-09-16T13:30:24.065297+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043166400000000001,"stat_sum":{"num_bytes":10658027,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":13,"num_read":3521,"num_read_kb":37836,"num_write":588,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053779+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:22.053779+0000","last_peered":"2024-09-16T13:30:22.053779+0000","last_clean":"2024-09-16T13:30:22.053779+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:22.053779+0000","last_undegraded":"2024-09-16T13:30:22.053779+0000","last_fullsized":"2024-09-16T13:30:22.053779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"136'333","reported_seq":2554,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062991+0000","last_change":"2024-09-16T13:30:24.062991+0000","last_active":"2024-09-16T13:30:24.062991+0000","last_peered":"2024-09-16T13:30:24.062991+0000","last_clean":"2024-09-16T13:30:24.062991+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:24.062991+0000","last_undegraded":"2024-09-16T13:30:24.062991+0000","last_fullsized":"2024-09-16T13:30:24.062991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.00406997,"stat_sum":{"num_bytes":2725750,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":14,"num_read":2287,"num_read_kb":15219,"num_write":409,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.5","version":"0'0","reported_seq":110,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.052704+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:22.052704+0000","last_peered":"2024-09-16T13:30:22.052704+0000","last_clean":"2024-09-16T13:30:22.052704+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:22.052704+0000","last_undegraded":"2024-09-16T13:30:22.052704+0000","last_fullsized":"2024-09-16T13:30:22.052704+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"135'755","reported_seq":4069,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.145407+0000","last_change":"2024-09-16T13:30:23.053355+0000","last_active":"2024-09-16T13:30:23.145407+0000","last_peered":"2024-09-16T13:30:23.145407+0000","last_clean":"2024-09-16T13:30:23.145407+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:23.145407+0000","last_undegraded":"2024-09-16T13:30:23.145407+0000","last_fullsized":"2024-09-16T13:30:23.145407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010668139999999999,"stat_sum":{"num_bytes":37983676,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":17,"num_read":4968,"num_read_kb":49563,"num_write":852,"num_write_kb":31546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.054139+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:22.054139+0000","last_peered":"2024-09-16T13:30:22.054139+0000","last_clean":"2024-09-16T13:30:22.054139+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:22.054139+0000","last_undegraded":"2024-09-16T13:30:22.054139+0000","last_fullsized":"2024-09-16T13:30:22.054139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"131'579","reported_seq":3407,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.050948+0000","last_change":"2024-09-16T13:30:23.051376+0000","last_active":"2024-09-16T13:30:23.050948+0000","last_peered":"2024-09-16T13:30:23.050948+0000","last_clean":"2024-09-16T13:30:23.050948+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:23.050948+0000","last_undegraded":"2024-09-16T13:30:23.050948+0000","last_fullsized":"2024-09-16T13:30:23.050948+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053497799999999997,"stat_sum":{"num_bytes":3711080,"num_objects":16,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":5,"num_read":6327,"num_read_kb":18344,"num_write":649,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.3","version":"0'0","reported_seq":119,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.053091+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:23.053091+0000","last_peered":"2024-09-16T13:30:23.053091+0000","last_clean":"2024-09-16T13:30:23.053091+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:23.053091+0000","last_undegraded":"2024-09-16T13:30:23.053091+0000","last_fullsized":"2024-09-16T13:30:23.053091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":53,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053956+0000","last_change":"2024-09-16T13:29:55.469146+0000","last_active":"2024-09-16T13:30:22.053956+0000","last_peered":"2024-09-16T13:30:22.053956+0000","last_clean":"2024-09-16T13:30:22.053956+0000","last_became_active":"2024-09-16T13:29:55.468831+0000","last_became_peered":"2024-09-16T13:29:55.468831+0000","last_unstale":"2024-09-16T13:30:22.053956+0000","last_undegraded":"2024-09-16T13:30:22.053956+0000","last_fullsized":"2024-09-16T13:30:22.053956+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:56.220967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"131'359","reported_seq":4481,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.051419+0000","last_change":"2024-09-16T13:30:23.051855+0000","last_active":"2024-09-16T13:30:23.051419+0000","last_peered":"2024-09-16T13:30:23.051419+0000","last_clean":"2024-09-16T13:30:23.051419+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:23.051419+0000","last_undegraded":"2024-09-16T13:30:23.051419+0000","last_fullsized":"2024-09-16T13:30:23.051419+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071440999999999996,"stat_sum":{"num_bytes":4882523,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":6,"num_read":4247,"num_read_kb":35946,"num_write":389,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.0","version":"0'0","reported_seq":116,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053745+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:22.053745+0000","last_peered":"2024-09-16T13:30:22.053745+0000","last_clean":"2024-09-16T13:30:22.053745+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:22.053745+0000","last_undegraded":"2024-09-16T13:30:22.053745+0000","last_fullsized":"2024-09-16T13:30:22.053745+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":127,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.044949+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:22.044949+0000","last_peered":"2024-09-16T13:30:22.044949+0000","last_clean":"2024-09-16T13:30:22.044949+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:22.044949+0000","last_undegraded":"2024-09-16T13:30:22.044949+0000","last_fullsized":"2024-09-16T13:30:22.044949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"135'469","reported_seq":3381,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.057106+0000","last_change":"2024-09-16T13:30:23.057564+0000","last_active":"2024-09-16T13:30:23.057106+0000","last_peered":"2024-09-16T13:30:23.057106+0000","last_clean":"2024-09-16T13:30:23.057106+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:23.057106+0000","last_undegraded":"2024-09-16T13:30:23.057106+0000","last_fullsized":"2024-09-16T13:30:23.057106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0063987799999999997,"stat_sum":{"num_bytes":4372580,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":12,"num_read":4585,"num_read_kb":20423,"num_write":503,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.1","version":"105'2","reported_seq":120,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.076372+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:22.076372+0000","last_peered":"2024-09-16T13:30:22.076372+0000","last_clean":"2024-09-16T13:30:22.076372+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:22.076372+0000","last_undegraded":"2024-09-16T13:30:22.076372+0000","last_fullsized":"2024-09-16T13:30:22.076372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.055558+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:23.055558+0000","last_peered":"2024-09-16T13:30:23.055558+0000","last_clean":"2024-09-16T13:30:23.055558+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:23.055558+0000","last_undegraded":"2024-09-16T13:30:23.055558+0000","last_fullsized":"2024-09-16T13:30:23.055558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"135'527","reported_seq":3766,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.061170+0000","last_change":"2024-09-16T13:30:23.061170+0000","last_active":"2024-09-16T13:30:23.061170+0000","last_peered":"2024-09-16T13:30:23.061170+0000","last_clean":"2024-09-16T13:30:23.061170+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:23.061170+0000","last_undegraded":"2024-09-16T13:30:23.061170+0000","last_fullsized":"2024-09-16T13:30:23.061170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008255376,"stat_sum":{"num_bytes":24778457,"num_objects":32,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":19,"num_read":5126,"num_read_kb":40688,"num_write":509,"num_write_kb":30173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.2","version":"101'3","reported_seq":122,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.058224+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:23.058224+0000","last_peered":"2024-09-16T13:30:23.058224+0000","last_clean":"2024-09-16T13:30:23.058224+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:23.058224+0000","last_undegraded":"2024-09-16T13:30:23.058224+0000","last_fullsized":"2024-09-16T13:30:23.058224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":85,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.046034+0000","last_change":"2024-09-16T13:29:55.466745+0000","last_active":"2024-09-16T13:30:22.046034+0000","last_peered":"2024-09-16T13:30:22.046034+0000","last_clean":"2024-09-16T13:30:22.046034+0000","last_became_active":"2024-09-16T13:29:55.466418+0000","last_became_peered":"2024-09-16T13:29:55.466418+0000","last_unstale":"2024-09-16T13:30:22.046034+0000","last_undegraded":"2024-09-16T13:30:22.046034+0000","last_fullsized":"2024-09-16T13:30:22.046034+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:33.757459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":83,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.055692+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:23.055692+0000","last_peered":"2024-09-16T13:30:23.055692+0000","last_clean":"2024-09-16T13:30:23.055692+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:23.055692+0000","last_undegraded":"2024-09-16T13:30:23.055692+0000","last_fullsized":"2024-09-16T13:30:23.055692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"135'586","reported_seq":2427,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.110516+0000","last_change":"2024-09-16T13:30:23.110698+0000","last_active":"2024-09-16T13:30:23.110516+0000","last_peered":"2024-09-16T13:30:23.110516+0000","last_clean":"2024-09-16T13:30:23.110516+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:23.110516+0000","last_undegraded":"2024-09-16T13:30:23.110516+0000","last_fullsized":"2024-09-16T13:30:23.110516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0018664179999999999,"stat_sum":{"num_bytes":6761791,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":10,"num_read":3481,"num_read_kb":17248,"num_write":719,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.8","version":"136'1216","reported_seq":5574,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062550+0000","last_change":"2024-09-16T13:30:24.062853+0000","last_active":"2024-09-16T13:30:24.062550+0000","last_peered":"2024-09-16T13:30:24.062550+0000","last_clean":"2024-09-16T13:30:24.062550+0000","last_became_active":"2024-09-16T13:29:55.466349+0000","last_became_peered":"2024-09-16T13:29:55.466349+0000","last_unstale":"2024-09-16T13:30:24.062550+0000","last_undegraded":"2024-09-16T13:30:24.062550+0000","last_fullsized":"2024-09-16T13:30:24.062550+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:08.619344+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0022176219999999998,"stat_sum":{"num_bytes":6787085,"num_objects":24,"num_object_clones":7,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":9411,"num_read_kb":25459,"num_write":1800,"num_write_kb":11389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.9","version":"134'364","reported_seq":4715,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.069182+0000","last_change":"2024-09-16T13:30:24.069630+0000","last_active":"2024-09-16T13:30:24.069182+0000","last_peered":"2024-09-16T13:30:24.069182+0000","last_clean":"2024-09-16T13:30:24.069182+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:24.069182+0000","last_undegraded":"2024-09-16T13:30:24.069182+0000","last_fullsized":"2024-09-16T13:30:24.069182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042653789999999997,"stat_sum":{"num_bytes":16751805,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":5819,"num_read_kb":30041,"num_write":423,"num_write_kb":9465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.a","version":"135'524","reported_seq":4397,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.109351+0000","last_change":"2024-09-16T13:30:23.109351+0000","last_active":"2024-09-16T13:30:23.109351+0000","last_peered":"2024-09-16T13:30:23.109351+0000","last_clean":"2024-09-16T13:30:23.109351+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:23.109351+0000","last_undegraded":"2024-09-16T13:30:23.109351+0000","last_fullsized":"2024-09-16T13:30:23.109351+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046299500000000001,"stat_sum":{"num_bytes":20564596,"num_objects":27,"num_object_clones":7,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":15,"num_read":5807,"num_read_kb":43478,"num_write":559,"num_write_kb":25113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.b","version":"135'409","reported_seq":5407,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.058063+0000","last_change":"2024-09-16T13:30:23.058063+0000","last_active":"2024-09-16T13:30:23.058063+0000","last_peered":"2024-09-16T13:30:23.058063+0000","last_clean":"2024-09-16T13:30:23.058063+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:30:23.058063+0000","last_undegraded":"2024-09-16T13:30:23.058063+0000","last_fullsized":"2024-09-16T13:30:23.058063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0060088980000000004,"stat_sum":{"num_bytes":26173352,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":15,"num_read":5830,"num_read_kb":40470,"num_write":490,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.c","version":"134'467","reported_seq":5358,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.067814+0000","last_change":"2024-09-16T13:30:24.068061+0000","last_active":"2024-09-16T13:30:24.067814+0000","last_peered":"2024-09-16T13:30:24.067814+0000","last_clean":"2024-09-16T13:30:24.067814+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:24.067814+0000","last_undegraded":"2024-09-16T13:30:24.067814+0000","last_fullsized":"2024-09-16T13:30:24.067814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028167420000000001,"stat_sum":{"num_bytes":1917021,"num_objects":28,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":17,"num_read":6390,"num_read_kb":25158,"num_write":538,"num_write_kb":8621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.d","version":"135'435","reported_seq":3083,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.060583+0000","last_change":"2024-09-16T13:30:23.060823+0000","last_active":"2024-09-16T13:30:23.060583+0000","last_peered":"2024-09-16T13:30:23.060583+0000","last_clean":"2024-09-16T13:30:23.060583+0000","last_became_active":"2024-09-16T13:29:55.469723+0000","last_became_peered":"2024-09-16T13:29:55.469723+0000","last_unstale":"2024-09-16T13:30:23.060583+0000","last_undegraded":"2024-09-16T13:30:23.060583+0000","last_fullsized":"2024-09-16T13:30:23.060583+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.171374+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0086065069999999994,"stat_sum":{"num_bytes":10416100,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":18,"num_read":2950,"num_read_kb":22131,"num_write":456,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.e","version":"133'353","reported_seq":1727,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.068107+0000","last_change":"2024-09-16T13:30:24.068356+0000","last_active":"2024-09-16T13:30:24.068107+0000","last_peered":"2024-09-16T13:30:24.068107+0000","last_clean":"2024-09-16T13:30:24.068107+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:24.068107+0000","last_undegraded":"2024-09-16T13:30:24.068107+0000","last_fullsized":"2024-09-16T13:30:24.068107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030918830000000001,"stat_sum":{"num_bytes":11438007,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":22,"num_read":1718,"num_read_kb":17535,"num_write":412,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.f","version":"135'810","reported_seq":3329,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.075190+0000","last_change":"2024-09-16T13:30:23.075355+0000","last_active":"2024-09-16T13:30:23.075190+0000","last_peered":"2024-09-16T13:30:23.075190+0000","last_clean":"2024-09-16T13:30:23.075190+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:23.075190+0000","last_undegraded":"2024-09-16T13:30:23.075190+0000","last_fullsized":"2024-09-16T13:30:23.075190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0019899639999999999,"stat_sum":{"num_bytes":10204540,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":19,"num_read":5466,"num_read_kb":14688,"num_write":1104,"num_write_kb":7902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"a"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":384365859,"num_objects":785,"num_object_clones":158,"num_object_copies":1570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":431,"num_read":163210,"num_read_kb":801757,"num_write":21148,"num_write_kb":460795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24719360,"data_stored":290921803,"data_compressed":4917019,"data_compressed_allocated":20754432,"data_compressed_original":288826722,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":168,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509472,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305072,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066768,"statfs":{"total":96636764160,"available":96324370432,"internally_reserved":0,"allocated":3633152,"data_stored":20707763,"data_compressed":358216,"data_compressed_allocated":1683456,"data_compressed_original":19722240,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.376},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.33600000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509474,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307152,"kb_used_data":5624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94064688,"statfs":{"total":96636764160,"available":96322240512,"internally_reserved":0,"allocated":5758976,"data_stored":53851099,"data_compressed":733120,"data_compressed_allocated":3489792,"data_compressed_original":52678463,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.503},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.441}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":4272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":4374528,"data_stored":26459453,"data_compressed":743740,"data_compressed_allocated":2248704,"data_compressed_original":25367410,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33000000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]}]},{"osd":3,"up_from":15,"seq":64424509472,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309808,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062032,"statfs":{"total":96636764160,"available":96319520768,"internally_reserved":0,"allocated":4222976,"data_stored":20649956,"data_compressed":682178,"data_compressed_allocated":1867776,"data_compressed_original":19460096,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.307}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.318}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.436}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.42499999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301604,"kb_used_data":4244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070236,"statfs":{"total":96636764160,"available":96327921664,"internally_reserved":0,"allocated":4345856,"data_stored":39394224,"data_compressed":558072,"data_compressed_allocated":2383872,"data_compressed_original":38396786,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.439},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.307}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.373},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.35499999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48899999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542176,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308064,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063776,"statfs":{"total":96636764160,"available":96321306624,"internally_reserved":0,"allocated":2441216,"data_stored":5926756,"data_compressed":304896,"data_compressed_allocated":589824,"data_compressed_original":4980736,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77200000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.63100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312844,"kb_used_data":7160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94058996,"statfs":{"total":96636764160,"available":96316411904,"internally_reserved":0,"allocated":7331840,"data_stored":68108616,"data_compressed":1084718,"data_compressed_allocated":4739072,"data_compressed_original":66744320,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.216},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.312},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.25},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.41199999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.438},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.40400000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542176,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303336,"kb_used_data":5976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068504,"statfs":{"total":96636764160,"available":96326148096,"internally_reserved":0,"allocated":6119424,"data_stored":65158472,"data_compressed":634747,"data_compressed_allocated":4083712,"data_compressed_original":64106303,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.315},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.37}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.307},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4448256,"data_stored":63752865,"data_compressed":629148,"data_compressed_allocated":4046848,"data_compressed_original":63532863,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5513216,"data_stored":66731231,"data_compressed":1002998,"data_compressed_allocated":4644864,"data_compressed_original":66277376,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":5075455,"data_compressed":280382,"data_compressed_allocated":565248,"data_compressed_original":4943872,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":19716950,"data_compressed":660309,"data_compressed_allocated":1830912,"data_compressed_original":19353600,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":25370147,"data_compressed":715541,"data_compressed_allocated":2203648,"data_compressed_original":25101170,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3997696,"data_stored":52473548,"data_compressed":718924,"data_compressed_allocated":3440640,"data_compressed_original":52166463,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2727936,"data_stored":38506096,"data_compressed":557931,"data_compressed_allocated":2375680,"data_compressed_original":38318962,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1970176,"data_stored":19295511,"data_compressed":351786,"data_compressed_allocated":1646592,"data_compressed_original":19132416,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:26.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:26.439 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.439 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:26.495 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:26.492+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:26.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:26.586 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.586 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:26.595 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:26.593+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:26.625 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:26.640 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:26.640 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:26.655 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:26.670 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:26.670 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:26.685 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:26.693 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:26.693 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:26.696 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:26.693+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:26.700 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:26.701 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:26.702 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:26.715 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:26.715 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:26.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:26.735 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:26.736 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:26.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:26.747 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T13:30:26.052502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384956414,"num_objects":794,"num_object_clones":158,"num_object_copies":1588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":434,"num_read":163276,"num_read_kb":801812,"num_write":21253,"num_write_kb":462427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":13,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17638,"ondisk_log_size":17638,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449520,"kb_used_data":37332,"kb_used_omap":13,"kb_used_meta":2412018,"kb_avail":752525200,"statfs":{"total":773094113280,"available":770585804800,"internally_reserved":0,"allocated":38227968,"data_stored":300256339,"data_compressed":5099687,"data_compressed_allocated":21086208,"data_compressed_original":291456354,"omap_allocated":13632,"internal_metadata":2469907136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-272074323,"num_objects":-126,"num_object_clones":-35,"num_object_copies":-252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-126,"num_whiteouts":-23,"num_read":1369,"num_read_kb":1166,"num_write":212,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009943"},"pg_stats":[{"pgid":"4.1f","version":"135'338","reported_seq":2102,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.161651+0000","last_change":"2024-09-16T13:30:23.109144+0000","last_active":"2024-09-16T13:30:23.161651+0000","last_peered":"2024-09-16T13:30:23.161651+0000","last_clean":"2024-09-16T13:30:23.161651+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:23.161651+0000","last_undegraded":"2024-09-16T13:30:23.161651+0000","last_fullsized":"2024-09-16T13:30:23.161651+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032560199999999998,"stat_sum":{"num_bytes":5027510,"num_objects":23,"num_object_clones":2,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":2710,"num_read_kb":20014,"num_write":336,"num_write_kb":9428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1e","version":"135'1071","reported_seq":11739,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.140135+0000","last_change":"2024-09-16T13:30:23.058593+0000","last_active":"2024-09-16T13:30:23.140135+0000","last_peered":"2024-09-16T13:30:23.140135+0000","last_clean":"2024-09-16T13:30:23.140135+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:23.140135+0000","last_undegraded":"2024-09-16T13:30:23.140135+0000","last_fullsized":"2024-09-16T13:30:23.140135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061878769999999996,"stat_sum":{"num_bytes":3083383,"num_objects":23,"num_object_clones":4,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":14,"num_read":14968,"num_read_kb":31050,"num_write":1650,"num_write_kb":9687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1d","version":"136'415","reported_seq":1489,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.064004+0000","last_change":"2024-09-16T13:30:24.064276+0000","last_active":"2024-09-16T13:30:24.064004+0000","last_peered":"2024-09-16T13:30:24.064004+0000","last_clean":"2024-09-16T13:30:24.064004+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:24.064004+0000","last_undegraded":"2024-09-16T13:30:24.064004+0000","last_fullsized":"2024-09-16T13:30:24.064004+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0055176310000000003,"stat_sum":{"num_bytes":3000237,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":14,"num_read":1819,"num_read_kb":7961,"num_write":429,"num_write_kb":5828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.1c","version":"135'484","reported_seq":4451,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.093897+0000","last_change":"2024-09-16T13:30:23.074030+0000","last_active":"2024-09-16T13:30:23.093897+0000","last_peered":"2024-09-16T13:30:23.093897+0000","last_clean":"2024-09-16T13:30:23.093897+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:23.093897+0000","last_undegraded":"2024-09-16T13:30:23.093897+0000","last_fullsized":"2024-09-16T13:30:23.093897+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060536599999999998,"stat_sum":{"num_bytes":5225978,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":8,"num_read":7571,"num_read_kb":26873,"num_write":490,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.1b","version":"136'468","reported_seq":2303,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.072365+0000","last_change":"2024-09-16T13:30:24.072511+0000","last_active":"2024-09-16T13:30:24.072365+0000","last_peered":"2024-09-16T13:30:24.072365+0000","last_clean":"2024-09-16T13:30:24.072365+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:24.072365+0000","last_undegraded":"2024-09-16T13:30:24.072365+0000","last_fullsized":"2024-09-16T13:30:24.072365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.0071583360000000004,"stat_sum":{"num_bytes":4883732,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":9,"num_read":3240,"num_read_kb":15629,"num_write":533,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.1a","version":"136'398","reported_seq":4170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.155249+0000","last_change":"2024-09-16T13:30:24.067774+0000","last_active":"2024-09-16T13:30:24.155249+0000","last_peered":"2024-09-16T13:30:24.155249+0000","last_clean":"2024-09-16T13:30:24.155249+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:24.155249+0000","last_undegraded":"2024-09-16T13:30:24.155249+0000","last_fullsized":"2024-09-16T13:30:24.155249+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0026119400000000001,"stat_sum":{"num_bytes":11030449,"num_objects":23,"num_object_clones":1,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":4026,"num_read_kb":16476,"num_write":437,"num_write_kb":6964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.19","version":"136'386","reported_seq":3428,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.061210+0000","last_change":"2024-09-16T13:30:24.061529+0000","last_active":"2024-09-16T13:30:24.061210+0000","last_peered":"2024-09-16T13:30:24.061210+0000","last_clean":"2024-09-16T13:30:24.061210+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:24.061210+0000","last_undegraded":"2024-09-16T13:30:24.061210+0000","last_fullsized":"2024-09-16T13:30:24.061210+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.003079933,"stat_sum":{"num_bytes":26423659,"num_objects":29,"num_object_clones":6,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":14,"num_read":3419,"num_read_kb":15850,"num_write":388,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.18","version":"136'371","reported_seq":3365,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.067186+0000","last_change":"2024-09-16T13:30:24.067410+0000","last_active":"2024-09-16T13:30:24.067186+0000","last_peered":"2024-09-16T13:30:24.067186+0000","last_clean":"2024-09-16T13:30:24.067186+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:24.067186+0000","last_undegraded":"2024-09-16T13:30:24.067186+0000","last_fullsized":"2024-09-16T13:30:24.067186+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.002300182,"stat_sum":{"num_bytes":23217310,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":10,"num_read":3439,"num_read_kb":32799,"num_write":387,"num_write_kb":30744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.17","version":"135'399","reported_seq":3188,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062538+0000","last_change":"2024-09-16T13:30:24.063099+0000","last_active":"2024-09-16T13:30:24.062538+0000","last_peered":"2024-09-16T13:30:24.062538+0000","last_clean":"2024-09-16T13:30:24.062538+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:24.062538+0000","last_undegraded":"2024-09-16T13:30:24.062538+0000","last_fullsized":"2024-09-16T13:30:24.062538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042827439999999998,"stat_sum":{"num_bytes":9490556,"num_objects":28,"num_object_clones":5,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":20,"num_read":4459,"num_read_kb":29774,"num_write":362,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.16","version":"135'649","reported_seq":2805,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.064071+0000","last_change":"2024-09-16T13:30:23.064071+0000","last_active":"2024-09-16T13:30:23.064071+0000","last_peered":"2024-09-16T13:30:23.064071+0000","last_clean":"2024-09-16T13:30:23.064071+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:23.064071+0000","last_undegraded":"2024-09-16T13:30:23.064071+0000","last_fullsized":"2024-09-16T13:30:23.064071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011372347999999999,"stat_sum":{"num_bytes":968704,"num_objects":12,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":8,"num_read":4642,"num_read_kb":15931,"num_write":788,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.15","version":"136'808","reported_seq":4060,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062363+0000","last_change":"2024-09-16T13:30:24.062602+0000","last_active":"2024-09-16T13:30:24.062363+0000","last_peered":"2024-09-16T13:30:24.062363+0000","last_clean":"2024-09-16T13:30:24.062363+0000","last_became_active":"2024-09-16T13:29:55.466429+0000","last_became_peered":"2024-09-16T13:29:55.466429+0000","last_unstale":"2024-09-16T13:30:24.062363+0000","last_undegraded":"2024-09-16T13:30:24.062363+0000","last_fullsized":"2024-09-16T13:30:24.062363+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:31.786154+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001939035,"stat_sum":{"num_bytes":50683772,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":15,"num_read":5783,"num_read_kb":34352,"num_write":1210,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.14","version":"132'466","reported_seq":3420,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.061607+0000","last_change":"2024-09-16T13:30:24.061928+0000","last_active":"2024-09-16T13:30:24.061607+0000","last_peered":"2024-09-16T13:30:24.061607+0000","last_clean":"2024-09-16T13:30:24.061607+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:24.061607+0000","last_undegraded":"2024-09-16T13:30:24.061607+0000","last_fullsized":"2024-09-16T13:30:24.061607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031719180000000001,"stat_sum":{"num_bytes":5214933,"num_objects":33,"num_object_clones":9,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":19,"num_read":4915,"num_read_kb":20477,"num_write":499,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.13","version":"136'359","reported_seq":1940,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.068892+0000","last_change":"2024-09-16T13:30:24.069122+0000","last_active":"2024-09-16T13:30:24.068892+0000","last_peered":"2024-09-16T13:30:24.068892+0000","last_clean":"2024-09-16T13:30:24.068892+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:24.068892+0000","last_undegraded":"2024-09-16T13:30:24.068892+0000","last_fullsized":"2024-09-16T13:30:24.068892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0039196500000000002,"stat_sum":{"num_bytes":29790726,"num_objects":28,"num_object_clones":7,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":17,"num_read":1521,"num_read_kb":22420,"num_write":346,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.12","version":"135'794","reported_seq":4142,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.164081+0000","last_change":"2024-09-16T13:30:23.051440+0000","last_active":"2024-09-16T13:30:23.164081+0000","last_peered":"2024-09-16T13:30:23.164081+0000","last_clean":"2024-09-16T13:30:23.164081+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:23.164081+0000","last_undegraded":"2024-09-16T13:30:23.164081+0000","last_fullsized":"2024-09-16T13:30:23.164081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063509500000000002,"stat_sum":{"num_bytes":348190,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":10,"num_read":8606,"num_read_kb":20548,"num_write":964,"num_write_kb":8823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.11","version":"133'748","reported_seq":4756,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.059003+0000","last_change":"2024-09-16T13:30:24.059301+0000","last_active":"2024-09-16T13:30:24.059003+0000","last_peered":"2024-09-16T13:30:24.059003+0000","last_clean":"2024-09-16T13:30:24.059003+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:24.059003+0000","last_undegraded":"2024-09-16T13:30:24.059003+0000","last_fullsized":"2024-09-16T13:30:24.059003+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088673099999999998,"stat_sum":{"num_bytes":3740890,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":15,"num_read":5903,"num_read_kb":16215,"num_write":942,"num_write_kb":25099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.10","version":"135'731","reported_seq":3766,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.058478+0000","last_change":"2024-09-16T13:30:24.058954+0000","last_active":"2024-09-16T13:30:24.058478+0000","last_peered":"2024-09-16T13:30:24.058478+0000","last_clean":"2024-09-16T13:30:24.058478+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:24.058478+0000","last_undegraded":"2024-09-16T13:30:24.058478+0000","last_fullsized":"2024-09-16T13:30:24.058478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058177299999999995,"stat_sum":{"num_bytes":2109440,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":13,"num_read":8256,"num_read_kb":21161,"num_write":987,"num_write_kb":9535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.7","version":"0'0","reported_seq":145,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.052852+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:22.052852+0000","last_peered":"2024-09-16T13:30:22.052852+0000","last_clean":"2024-09-16T13:30:22.052852+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:22.052852+0000","last_undegraded":"2024-09-16T13:30:22.052852+0000","last_fullsized":"2024-09-16T13:30:22.052852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"133'487","reported_seq":3543,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.065297+0000","last_change":"2024-09-16T13:30:24.065569+0000","last_active":"2024-09-16T13:30:24.065297+0000","last_peered":"2024-09-16T13:30:24.065297+0000","last_clean":"2024-09-16T13:30:24.065297+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:24.065297+0000","last_undegraded":"2024-09-16T13:30:24.065297+0000","last_fullsized":"2024-09-16T13:30:24.065297+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043166400000000001,"stat_sum":{"num_bytes":10658027,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":13,"num_read":3521,"num_read_kb":37836,"num_write":588,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053779+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:22.053779+0000","last_peered":"2024-09-16T13:30:22.053779+0000","last_clean":"2024-09-16T13:30:22.053779+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:22.053779+0000","last_undegraded":"2024-09-16T13:30:22.053779+0000","last_fullsized":"2024-09-16T13:30:22.053779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"136'333","reported_seq":2554,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062991+0000","last_change":"2024-09-16T13:30:24.062991+0000","last_active":"2024-09-16T13:30:24.062991+0000","last_peered":"2024-09-16T13:30:24.062991+0000","last_clean":"2024-09-16T13:30:24.062991+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:24.062991+0000","last_undegraded":"2024-09-16T13:30:24.062991+0000","last_fullsized":"2024-09-16T13:30:24.062991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.00406997,"stat_sum":{"num_bytes":2725750,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":14,"num_read":2287,"num_read_kb":15219,"num_write":409,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"2.5","version":"0'0","reported_seq":110,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.052704+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:22.052704+0000","last_peered":"2024-09-16T13:30:22.052704+0000","last_clean":"2024-09-16T13:30:22.052704+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:22.052704+0000","last_undegraded":"2024-09-16T13:30:22.052704+0000","last_fullsized":"2024-09-16T13:30:22.052704+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"135'755","reported_seq":4069,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.145407+0000","last_change":"2024-09-16T13:30:23.053355+0000","last_active":"2024-09-16T13:30:23.145407+0000","last_peered":"2024-09-16T13:30:23.145407+0000","last_clean":"2024-09-16T13:30:23.145407+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:23.145407+0000","last_undegraded":"2024-09-16T13:30:23.145407+0000","last_fullsized":"2024-09-16T13:30:23.145407+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010668139999999999,"stat_sum":{"num_bytes":37983676,"num_objects":36,"num_object_clones":9,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":17,"num_read":4968,"num_read_kb":49563,"num_write":852,"num_write_kb":31546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.054139+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:22.054139+0000","last_peered":"2024-09-16T13:30:22.054139+0000","last_clean":"2024-09-16T13:30:22.054139+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:22.054139+0000","last_undegraded":"2024-09-16T13:30:22.054139+0000","last_fullsized":"2024-09-16T13:30:22.054139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"131'579","reported_seq":3407,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.050948+0000","last_change":"2024-09-16T13:30:23.051376+0000","last_active":"2024-09-16T13:30:23.050948+0000","last_peered":"2024-09-16T13:30:23.050948+0000","last_clean":"2024-09-16T13:30:23.050948+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:23.050948+0000","last_undegraded":"2024-09-16T13:30:23.050948+0000","last_fullsized":"2024-09-16T13:30:23.050948+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053497799999999997,"stat_sum":{"num_bytes":3711080,"num_objects":16,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":5,"num_read":6327,"num_read_kb":18344,"num_write":649,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.3","version":"0'0","reported_seq":119,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.053091+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:23.053091+0000","last_peered":"2024-09-16T13:30:23.053091+0000","last_clean":"2024-09-16T13:30:23.053091+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:23.053091+0000","last_undegraded":"2024-09-16T13:30:23.053091+0000","last_fullsized":"2024-09-16T13:30:23.053091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":53,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053956+0000","last_change":"2024-09-16T13:29:55.469146+0000","last_active":"2024-09-16T13:30:22.053956+0000","last_peered":"2024-09-16T13:30:22.053956+0000","last_clean":"2024-09-16T13:30:22.053956+0000","last_became_active":"2024-09-16T13:29:55.468831+0000","last_became_peered":"2024-09-16T13:29:55.468831+0000","last_unstale":"2024-09-16T13:30:22.053956+0000","last_undegraded":"2024-09-16T13:30:22.053956+0000","last_fullsized":"2024-09-16T13:30:22.053956+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:56.220967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"131'359","reported_seq":4481,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.051419+0000","last_change":"2024-09-16T13:30:23.051855+0000","last_active":"2024-09-16T13:30:23.051419+0000","last_peered":"2024-09-16T13:30:23.051419+0000","last_clean":"2024-09-16T13:30:23.051419+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:23.051419+0000","last_undegraded":"2024-09-16T13:30:23.051419+0000","last_fullsized":"2024-09-16T13:30:23.051419+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071440999999999996,"stat_sum":{"num_bytes":4882523,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":6,"num_read":4247,"num_read_kb":35946,"num_write":389,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.0","version":"0'0","reported_seq":116,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.053745+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:22.053745+0000","last_peered":"2024-09-16T13:30:22.053745+0000","last_clean":"2024-09-16T13:30:22.053745+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:22.053745+0000","last_undegraded":"2024-09-16T13:30:22.053745+0000","last_fullsized":"2024-09-16T13:30:22.053745+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":127,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.044949+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:22.044949+0000","last_peered":"2024-09-16T13:30:22.044949+0000","last_clean":"2024-09-16T13:30:22.044949+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:22.044949+0000","last_undegraded":"2024-09-16T13:30:22.044949+0000","last_fullsized":"2024-09-16T13:30:22.044949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"135'469","reported_seq":3381,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.057106+0000","last_change":"2024-09-16T13:30:23.057564+0000","last_active":"2024-09-16T13:30:23.057106+0000","last_peered":"2024-09-16T13:30:23.057106+0000","last_clean":"2024-09-16T13:30:23.057106+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:23.057106+0000","last_undegraded":"2024-09-16T13:30:23.057106+0000","last_fullsized":"2024-09-16T13:30:23.057106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0063987799999999997,"stat_sum":{"num_bytes":4372580,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":12,"num_read":4585,"num_read_kb":20423,"num_write":503,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.1","version":"105'2","reported_seq":120,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.076372+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:22.076372+0000","last_peered":"2024-09-16T13:30:22.076372+0000","last_clean":"2024-09-16T13:30:22.076372+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:22.076372+0000","last_undegraded":"2024-09-16T13:30:22.076372+0000","last_fullsized":"2024-09-16T13:30:22.076372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.055558+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:23.055558+0000","last_peered":"2024-09-16T13:30:23.055558+0000","last_clean":"2024-09-16T13:30:23.055558+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:23.055558+0000","last_undegraded":"2024-09-16T13:30:23.055558+0000","last_fullsized":"2024-09-16T13:30:23.055558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"135'527","reported_seq":3766,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.061170+0000","last_change":"2024-09-16T13:30:23.061170+0000","last_active":"2024-09-16T13:30:23.061170+0000","last_peered":"2024-09-16T13:30:23.061170+0000","last_clean":"2024-09-16T13:30:23.061170+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:23.061170+0000","last_undegraded":"2024-09-16T13:30:23.061170+0000","last_fullsized":"2024-09-16T13:30:23.061170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008255376,"stat_sum":{"num_bytes":24778457,"num_objects":32,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":19,"num_read":5126,"num_read_kb":40688,"num_write":509,"num_write_kb":30173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"3.2","version":"101'3","reported_seq":122,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.058224+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:23.058224+0000","last_peered":"2024-09-16T13:30:23.058224+0000","last_clean":"2024-09-16T13:30:23.058224+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:23.058224+0000","last_undegraded":"2024-09-16T13:30:23.058224+0000","last_fullsized":"2024-09-16T13:30:23.058224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":85,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":44,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T13:30:22.046034+0000","last_change":"2024-09-16T13:29:55.466745+0000","last_active":"2024-09-16T13:30:22.046034+0000","last_peered":"2024-09-16T13:30:22.046034+0000","last_clean":"2024-09-16T13:30:22.046034+0000","last_became_active":"2024-09-16T13:29:55.466418+0000","last_became_peered":"2024-09-16T13:29:55.466418+0000","last_unstale":"2024-09-16T13:30:22.046034+0000","last_undegraded":"2024-09-16T13:30:22.046034+0000","last_fullsized":"2024-09-16T13:30:22.046034+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:33.757459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":83,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.055692+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:23.055692+0000","last_peered":"2024-09-16T13:30:23.055692+0000","last_clean":"2024-09-16T13:30:23.055692+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:23.055692+0000","last_undegraded":"2024-09-16T13:30:23.055692+0000","last_fullsized":"2024-09-16T13:30:23.055692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"135'586","reported_seq":2427,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.110516+0000","last_change":"2024-09-16T13:30:23.110698+0000","last_active":"2024-09-16T13:30:23.110516+0000","last_peered":"2024-09-16T13:30:23.110516+0000","last_clean":"2024-09-16T13:30:23.110516+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:23.110516+0000","last_undegraded":"2024-09-16T13:30:23.110516+0000","last_fullsized":"2024-09-16T13:30:23.110516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0018664179999999999,"stat_sum":{"num_bytes":6761791,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":10,"num_read":3481,"num_read_kb":17248,"num_write":719,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.8","version":"136'1216","reported_seq":5574,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.062550+0000","last_change":"2024-09-16T13:30:24.062853+0000","last_active":"2024-09-16T13:30:24.062550+0000","last_peered":"2024-09-16T13:30:24.062550+0000","last_clean":"2024-09-16T13:30:24.062550+0000","last_became_active":"2024-09-16T13:29:55.466349+0000","last_became_peered":"2024-09-16T13:29:55.466349+0000","last_unstale":"2024-09-16T13:30:24.062550+0000","last_undegraded":"2024-09-16T13:30:24.062550+0000","last_fullsized":"2024-09-16T13:30:24.062550+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:08.619344+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0022176219999999998,"stat_sum":{"num_bytes":6787085,"num_objects":24,"num_object_clones":7,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":9,"num_read":9411,"num_read_kb":25459,"num_write":1800,"num_write_kb":11389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.9","version":"134'364","reported_seq":4715,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.069182+0000","last_change":"2024-09-16T13:30:24.069630+0000","last_active":"2024-09-16T13:30:24.069182+0000","last_peered":"2024-09-16T13:30:24.069182+0000","last_clean":"2024-09-16T13:30:24.069182+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:24.069182+0000","last_undegraded":"2024-09-16T13:30:24.069182+0000","last_fullsized":"2024-09-16T13:30:24.069182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042653789999999997,"stat_sum":{"num_bytes":16751805,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":11,"num_read":5819,"num_read_kb":30041,"num_write":423,"num_write_kb":9465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.a","version":"135'524","reported_seq":4397,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.109351+0000","last_change":"2024-09-16T13:30:23.109351+0000","last_active":"2024-09-16T13:30:23.109351+0000","last_peered":"2024-09-16T13:30:23.109351+0000","last_clean":"2024-09-16T13:30:23.109351+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:23.109351+0000","last_undegraded":"2024-09-16T13:30:23.109351+0000","last_fullsized":"2024-09-16T13:30:23.109351+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046299500000000001,"stat_sum":{"num_bytes":20564596,"num_objects":27,"num_object_clones":7,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":15,"num_read":5807,"num_read_kb":43478,"num_write":559,"num_write_kb":25113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.b","version":"135'409","reported_seq":5407,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.058063+0000","last_change":"2024-09-16T13:30:23.058063+0000","last_active":"2024-09-16T13:30:23.058063+0000","last_peered":"2024-09-16T13:30:23.058063+0000","last_clean":"2024-09-16T13:30:23.058063+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:30:23.058063+0000","last_undegraded":"2024-09-16T13:30:23.058063+0000","last_fullsized":"2024-09-16T13:30:23.058063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0060088980000000004,"stat_sum":{"num_bytes":26173352,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":15,"num_read":5830,"num_read_kb":40470,"num_write":490,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.c","version":"134'467","reported_seq":5358,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.067814+0000","last_change":"2024-09-16T13:30:24.068061+0000","last_active":"2024-09-16T13:30:24.067814+0000","last_peered":"2024-09-16T13:30:24.067814+0000","last_clean":"2024-09-16T13:30:24.067814+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:24.067814+0000","last_undegraded":"2024-09-16T13:30:24.067814+0000","last_fullsized":"2024-09-16T13:30:24.067814+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028167420000000001,"stat_sum":{"num_bytes":1917021,"num_objects":28,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":17,"num_read":6390,"num_read_kb":25158,"num_write":538,"num_write_kb":8621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.d","version":"135'435","reported_seq":3083,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.060583+0000","last_change":"2024-09-16T13:30:23.060823+0000","last_active":"2024-09-16T13:30:23.060583+0000","last_peered":"2024-09-16T13:30:23.060583+0000","last_clean":"2024-09-16T13:30:23.060583+0000","last_became_active":"2024-09-16T13:29:55.469723+0000","last_became_peered":"2024-09-16T13:29:55.469723+0000","last_unstale":"2024-09-16T13:30:23.060583+0000","last_undegraded":"2024-09-16T13:30:23.060583+0000","last_fullsized":"2024-09-16T13:30:23.060583+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.171374+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0086065069999999994,"stat_sum":{"num_bytes":10416100,"num_objects":33,"num_object_clones":7,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":18,"num_read":2950,"num_read_kb":22131,"num_write":456,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"a"}]},{"pgid":"4.e","version":"133'353","reported_seq":1727,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T13:30:24.068107+0000","last_change":"2024-09-16T13:30:24.068356+0000","last_active":"2024-09-16T13:30:24.068107+0000","last_peered":"2024-09-16T13:30:24.068107+0000","last_clean":"2024-09-16T13:30:24.068107+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:24.068107+0000","last_undegraded":"2024-09-16T13:30:24.068107+0000","last_fullsized":"2024-09-16T13:30:24.068107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030918830000000001,"stat_sum":{"num_bytes":11438007,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":22,"num_read":1718,"num_read_kb":17535,"num_write":412,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"b"}]},{"pgid":"4.f","version":"135'810","reported_seq":3329,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T13:30:23.075190+0000","last_change":"2024-09-16T13:30:23.075355+0000","last_active":"2024-09-16T13:30:23.075190+0000","last_peered":"2024-09-16T13:30:23.075190+0000","last_clean":"2024-09-16T13:30:23.075190+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:23.075190+0000","last_undegraded":"2024-09-16T13:30:23.075190+0000","last_fullsized":"2024-09-16T13:30:23.075190+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0019899639999999999,"stat_sum":{"num_bytes":10204540,"num_objects":36,"num_object_clones":10,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":19,"num_read":5466,"num_read_kb":14688,"num_write":1104,"num_write_kb":7902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"a"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":384365859,"num_objects":785,"num_object_clones":158,"num_object_copies":1570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":431,"num_read":163210,"num_read_kb":801757,"num_write":21148,"num_write_kb":460795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24719360,"data_stored":290921803,"data_compressed":4917019,"data_compressed_allocated":20754432,"data_compressed_original":288826722,"omap_allocated":0,"internal_metadata":0},"log_size":17558,"ondisk_log_size":17558,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":168,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509472,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305072,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066768,"statfs":{"total":96636764160,"available":96324370432,"internally_reserved":0,"allocated":3633152,"data_stored":20707763,"data_compressed":358216,"data_compressed_allocated":1683456,"data_compressed_original":19722240,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.376},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.33600000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509474,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307152,"kb_used_data":5624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94064688,"statfs":{"total":96636764160,"available":96322240512,"internally_reserved":0,"allocated":5758976,"data_stored":53851099,"data_compressed":733120,"data_compressed_allocated":3489792,"data_compressed_original":52678463,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.503},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.441}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301640,"kb_used_data":4272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070200,"statfs":{"total":96636764160,"available":96327884800,"internally_reserved":0,"allocated":4374528,"data_stored":26459453,"data_compressed":743740,"data_compressed_allocated":2248704,"data_compressed_original":25367410,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33000000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]}]},{"osd":3,"up_from":15,"seq":64424509472,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309808,"kb_used_data":4124,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062032,"statfs":{"total":96636764160,"available":96319520768,"internally_reserved":0,"allocated":4222976,"data_stored":20649956,"data_compressed":682178,"data_compressed_allocated":1867776,"data_compressed_original":19460096,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.307}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.318}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.436}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.42499999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301604,"kb_used_data":4244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070236,"statfs":{"total":96636764160,"available":96327921664,"internally_reserved":0,"allocated":4345856,"data_stored":39394224,"data_compressed":558072,"data_compressed_allocated":2383872,"data_compressed_original":38396786,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.439},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.307}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.373},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.35499999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48899999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542176,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308064,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063776,"statfs":{"total":96636764160,"available":96321306624,"internally_reserved":0,"allocated":2441216,"data_stored":5926756,"data_compressed":304896,"data_compressed_allocated":589824,"data_compressed_original":4980736,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.26800000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77200000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.63100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312844,"kb_used_data":7160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94058996,"statfs":{"total":96636764160,"available":96316411904,"internally_reserved":0,"allocated":7331840,"data_stored":68108616,"data_compressed":1084718,"data_compressed_allocated":4739072,"data_compressed_original":66744320,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.216},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.312},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.25},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.41199999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.438},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.40400000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542176,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303336,"kb_used_data":5976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068504,"statfs":{"total":96636764160,"available":96326148096,"internally_reserved":0,"allocated":6119424,"data_stored":65158472,"data_compressed":634747,"data_compressed_allocated":4083712,"data_compressed_original":64106303,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.315},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.37}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.307},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4448256,"data_stored":63752865,"data_compressed":629148,"data_compressed_allocated":4046848,"data_compressed_original":63532863,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5513216,"data_stored":66731231,"data_compressed":1002998,"data_compressed_allocated":4644864,"data_compressed_original":66277376,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":5075455,"data_compressed":280382,"data_compressed_allocated":565248,"data_compressed_original":4943872,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2547712,"data_stored":19716950,"data_compressed":660309,"data_compressed_allocated":1830912,"data_compressed_original":19353600,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2703360,"data_stored":25370147,"data_compressed":715541,"data_compressed_allocated":2203648,"data_compressed_original":25101170,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3997696,"data_stored":52473548,"data_compressed":718924,"data_compressed_allocated":3440640,"data_compressed_original":52166463,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2727936,"data_stored":38506096,"data_compressed":557931,"data_compressed_allocated":2375680,"data_compressed_original":38318962,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1970176,"data_stored":19295511,"data_compressed":351786,"data_compressed_allocated":1646592,"data_compressed_original":19132416,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:26.750 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T13:30:26.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:30:26.750 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T13:30:26.750 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:30:26.751 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:30:26.751 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T13:30:26.797 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:26.795+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:26.802 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:26.802 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:26.802 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:26.802 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:26.802 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:26.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:26.851 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:26.851 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "" 2024-09-16T13:30:26.851 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:26.899 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:26.897+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:26.950 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:26.950 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:26.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 141.65856596899999, 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54690382599999998, 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.951 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289201+0000", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289202+0000", 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.672e-06 2024-09-16T13:30:26.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289209+0000", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289218+0000", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4809999999999995e-06 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289227+0000", 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6510000000000007e-06 2024-09-16T13:30:26.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289279+0000", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2587000000000003e-05 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.289287+0000", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5059999999999998e-06 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834449+0000", 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.54516204000000001 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834642+0000", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019315599999999999 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.834841+0000", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019899800000000001 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.835419+0000", 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057850999999999996 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836036+0000", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061676500000000002 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836075+0000", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8727999999999999e-05 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.836104+0000", 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9448e-05 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.956 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.95664703400001, 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078950340000000004, 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 48 2024-09-16T13:30:26.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991120+0000", 2024-09-16T13:30:26.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991125+0000", 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1710000000000001e-06 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991133+0000", 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9969999999999998e-06 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991136+0000", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.661e-06 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996390+0000", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052546509999999999 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996471+0000", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0088999999999998e-05 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996594+0000", 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012320399999999999 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998458+0000", 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018642159999999999 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998932+0000", 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047405599999999999 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.998978+0000", 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5631999999999999e-05 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999015+0000", 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6927000000000001e-05 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.962 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.956624525, 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079848260000000004, 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 49 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:26.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991142+0000", 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2800000000000003e-07 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991145+0000", 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5170000000000002e-06 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991149+0000", 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0380000000000001e-06 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991151+0000", 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1500000000000002e-06 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996889+0000", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057376659999999998 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.996950+0000", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1612999999999994e-05 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997058+0000", 2024-09-16T13:30:26.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107926 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999071+0000", 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020122540000000002 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999077+0000", 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1800000000000001e-06 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999102+0000", 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5265e-05 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999127+0000", 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4888999999999998e-05 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.968 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.95657819799999, 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084568820000000006, 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 50 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:26.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991188+0000", 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991189+0000", 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991336+0000", 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000147658 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991343+0000", 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3260000000000001e-06 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991349+0000", 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2840000000000003e-06 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997345+0000", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059962469999999997 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997419+0000", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4110999999999996e-05 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.997552+0000", 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013285 2024-09-16T13:30:26.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999176+0000", 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016241999999999999 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999576+0000", 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039985299999999998 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999605+0000", 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9209000000000001e-05 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999645+0000", 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9758000000000001e-05 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.956411441, 2024-09-16T13:30:26.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0089646229999999997, 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 51 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991355+0000", 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991360+0000", 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2640000000000002e-06 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991364+0000", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9759999999999997e-06 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991367+0000", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0580000000000002e-06 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999218+0000", 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078517039999999993 2024-09-16T13:30:26.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999284+0000", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5029000000000004e-05 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999362+0000", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8577000000000003e-05 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000091+0000", 2024-09-16T13:30:26.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00072877399999999996 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000260+0000", 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016954700000000001 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000296+0000", 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5806999999999998e-05 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000320+0000", 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3501000000000001e-05 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.956396676, 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0092620919999999995, 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.980 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:26.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991370+0000", 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991373+0000", 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8480000000000001e-06 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991377+0000", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5430000000000004e-06 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.991381+0000", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1990000000000002e-06 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999672+0000", 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082913290000000001 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999757+0000", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5350000000000001e-05 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:05.999894+0000", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013633100000000001 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000383+0000", 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048957700000000005 2024-09-16T13:30:26.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000554+0000", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017101400000000001 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000599+0000", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4734000000000002e-05 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.000632+0000", 2024-09-16T13:30:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2957000000000003e-05 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.61107435900001, 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0094744110000000003, 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.986 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 73 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336692+0000", 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336693+0000", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1799999999999995e-07 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336697+0000", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1439999999999996e-06 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336703+0000", 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:30:26.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336708+0000", 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342269+0000", 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055609199999999996 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342311+0000", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1922000000000003e-05 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342389+0000", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8046999999999993e-05 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344729+0000", 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023403460000000001 2024-09-16T13:30:26.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346092+0000", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001363047 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346133+0000", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0883999999999997e-05 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.346167+0000", 2024-09-16T13:30:26.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3702999999999999e-05 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.611036681, 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011856807, 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:26.995 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 74 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:26.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336730+0000", 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336734+0000", 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5090000000000002e-06 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:26.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336741+0000", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336747+0000", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342850+0000", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061035869999999997 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:26.998 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.342929+0000", 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8707000000000004e-05 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343048+0000", 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011928 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345066+0000", 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020178729999999999 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:26.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348516+0000", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034500630000000002 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348554+0000", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8146999999999999e-05 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.348587+0000", 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2225999999999999e-05 2024-09-16T13:30:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.61101386799999, 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014675855, 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:27.001 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336753+0000", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4000000000000002e-07 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336814+0000", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0851999999999999e-05 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336825+0000", 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.137e-05 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336833+0000", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4209999999999998e-06 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343340+0000", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0065075319999999999 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343417+0000", 2024-09-16T13:30:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6126999999999998e-05 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.343537+0000", 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120288 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345374+0000", 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018368130000000001 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351351+0000", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059775679999999999 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351395+0000", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4149e-05 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351429+0000", 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3195000000000003e-05 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4181.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 140.61092865500001, 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014672385, 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4181", 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1399189697", 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 76 2024-09-16T13:30:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336838+0000", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8200000000000001e-07 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336841+0000", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336845+0000", 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2930000000000002e-06 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.336847+0000", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3439999999999999e-06 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344278+0000", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074307790000000002 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344356+0000", 2024-09-16T13:30:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7503000000000007e-05 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0", 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.344476+0000", 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120506 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.345396+0000", 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092000800000000005 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351462+0000", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060661270000000002 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351491+0000", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8583000000000001e-05 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:06.351510+0000", 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9516000000000001e-05 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.012 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3216 4.15 e38/32 4:ae228924:::rbd_data.11a976a8123d.0000000000000004:head v 38'12, mlcod=38'9)", 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 112.30434973, 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017028913999999999, 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643417+0000", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.643418+0000", 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3820000000000001e-06 2024-09-16T13:30:27.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645088+0000", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016699060000000001 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645113+0000", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5057000000000001e-05 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645124+0000", 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0876000000000001e-05 2024-09-16T13:30:27.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645176+0000", 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2122e-05 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.645196+0000", 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9857e-05 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660413+0000", 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015217424 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.660446+0000", 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2289999999999997e-05 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3297 4.17 e38/32 4:ecb3b2db:::rbd_data.11a976a8123d.0000000000000002:head v 38'4, mlcod=37'1)", 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:27.017 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 112.278441496, 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028121987000000001, 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669325+0000", 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.669326+0000", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673446+0000", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041202169999999998 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673464+0000", 2024-09-16T13:30:27.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7875000000000001e-05 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673471+0000", 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8560000000000001e-06 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673524+0000", 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3467999999999997e-05 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.673554+0000", 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9478000000000001e-05 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697404+0000", 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023850268000000001 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.697447+0000", 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3170000000000002e-05 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.021 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:3537 4.15 e38/32 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head v 38'13, mlcod=38'11)", 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 112.20967634100001, 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018984896000000001, 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738090+0000", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.738092+0000", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4890000000000001e-06 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739125+0000", 2024-09-16T13:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010333009999999999 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739145+0000", 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9650999999999999e-05 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739152+0000", 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4479999999999997e-06 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739206+0000", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4033999999999998e-05 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.739232+0000", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5955000000000001e-05 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757020+0000", 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017788409000000002 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.757075+0000", 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.15 69/67 [PushOp(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7, version: 39'23, data_included: [0~622592,655360~98304,786432~229376,1048576~32768,1179648~65536], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(4:a82e7e0a:::rbd_data.11a976a8123d.000000000000000d:7@39'23, size: 10107276, copy_subset: [0~10107276], clone_subset: {}, snapset: 7=[]:{7=[7]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1245184, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 77.708133524000004, 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010174043000000001, 2024-09-16T13:30:27.026 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239633+0000", 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239634+0000", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6599999999999995e-07 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239956+0000", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032267500000000002 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.239979+0000", 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2793e-05 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.240093+0000", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000114027 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245007+0000", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049133909999999996 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.245013+0000", 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0419999999999999e-06 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.249807+0000", 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047943489999999998 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:67814 4.17 e80/66 4:e82035da:::rbd_data.11a97689809e.0000000000000019:head v 80'229, mlcod=80'227)", 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 63.513208204999998, 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014475228999999999, 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.030 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434558+0000", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.434560+0000", 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.178e-06 2024-09-16T13:30:27.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436955+0000", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023957589999999999 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436971+0000", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6093999999999999e-05 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.436978+0000", 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3149999999999997e-06 2024-09-16T13:30:27.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437025+0000", 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7466999999999998e-05 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.437047+0000", 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1902000000000001e-05 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449004+0000", 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011956823 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:23.449034+0000", 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9691000000000001e-05 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:88643 4.15 e87/70 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:head v 87'391, mlcod=87'389)", 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 54.357331170000002, 2024-09-16T13:30:27.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014175162999999999, 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590435+0000", 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.590437+0000", 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2750000000000001e-06 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591254+0000", 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081713399999999996 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591274+0000", 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0598000000000002e-05 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591284+0000", 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7410000000000004e-06 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591333+0000", 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9233000000000001e-05 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.591358+0000", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4456000000000001e-05 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604561+0000", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013203157 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:32.604611+0000", 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9568999999999999e-05 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:90394 4.1e e88/69 4:7bd73bb6:::rbd_data.11a93542739e.0000000000000007:head v 88'908, mlcod=85'905)", 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 53.078379339999998, 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013638523, 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869387+0000", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.869388+0000", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0499999999999999e-07 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871155+0000", 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001767795 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871171+0000", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5372000000000001e-05 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871179+0000", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2390000000000002e-06 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871241+0000", 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1598000000000002e-05 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.871267+0000", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061e-05 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.882977+0000", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011710399 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:33.883026+0000", 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8754000000000001e-05 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:99272 4.17 e95/66 4:ec840ba6:::rbd_data.11a97e9c0852.0000000000000004:head v 95'273, mlcod=94'271)", 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.303024633, 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019894543000000001, 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644742+0000", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.644743+0000", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.09e-07 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647864+0000", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031208569999999999 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647883+0000", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9746999999999999e-05 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647890+0000", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.63e-06 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647933+0000", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.647951+0000", 2024-09-16T13:30:27.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8726999999999999e-05 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664596+0000", 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016645138 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.664636+0000", 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0025999999999999e-05 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100152 4.15 e95/70 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head v 95'426, mlcod=92'424)", 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.205754212000002, 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022158348000000001, 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742012+0000", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742014+0000", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8139999999999999e-06 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743570+0000", 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015555129999999999 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743600+0000", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0142999999999999e-05 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743613+0000", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2913e-05 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743725+0000", 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112077 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743763+0000", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8241000000000002e-05 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764122+0000", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020358916000000001 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764171+0000", 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8730999999999998e-05 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100157 4.11 e95/69 4:8b32c473:::rbd_data.11a97e9c0852.000000000000000a:head v 95'602, mlcod=95'600)", 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.204072959000001, 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023662931000000002, 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743694+0000", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.743695+0000", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.215e-06 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748904+0000", 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052095939999999997 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748920+0000", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5736999999999999e-05 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748928+0000", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.542e-06 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.748996+0000", 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8547000000000006e-05 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.749018+0000", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767330+0000", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018311770000000002 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.767357+0000", 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6738e-05 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:27.055 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:27.055 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:27.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:27.057 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:27.053+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:27.157 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:27.155+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:27.258 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:27.257+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:27.360 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:27.358+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6344 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.21400699599999, 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10053079199999999, 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6344 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165665+0000", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.165668+0000", 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2759999999999999e-06 2024-09-16T13:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171091+0000", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054236789999999998 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171118+0000", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6707999999999999e-05 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171126+0000", 2024-09-16T13:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9799999999999998e-06 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171184+0000", 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8022000000000001e-05 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171328+0000", 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014405200000000001 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.172701+0000", 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013725619999999999 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.250918+0000", 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078217632999999995 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266075+0000", 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015156726000000001 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266142+0000", 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6951000000000003e-05 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266196+0000", 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4203000000000003e-05 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:6345 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head [truncate 7618367] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.208531431, 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095265840000000004, 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6345 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171141+0000", 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171142+0000", 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.102e-06 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171145+0000", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1049999999999999e-06 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171149+0000", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8310000000000003e-06 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.171156+0000", 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6420000000000007e-06 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242794+0000", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071637135000000005 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.242886+0000", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2354000000000004e-05 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.243067+0000", 2024-09-16T13:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018092800000000001 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.251364+0000", 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082969010000000006 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266357+0000", 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014992757000000001 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266384+0000", 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7223e-05 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.266407+0000", 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2861999999999999e-05 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:7703 4.3 4:c18a8bd3:::rbd_data.11a91a08d9ca.00000000000000d1:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.448204624, 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093004276999999996, 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7703 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931468+0000", 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931469+0000", 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.78e-07 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931473+0000", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8539999999999996e-06 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931485+0000", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824999999999999e-05 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931491+0000", 2024-09-16T13:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931526+0000", 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5148999999999998e-05 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.931606+0000", 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9962000000000005e-05 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.936244+0000", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004638273 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024298+0000", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088053751 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024410+0000", 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011181 2024-09-16T13:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024448+0000", 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7753999999999998e-05 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024472+0000", 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4091999999999999e-05 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:7703 4.3 e41/32 ondisk, result = 0)", 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.431666499, 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076295478999999999, 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948006+0000", 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948007+0000", 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999998e-07 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948011+0000", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948025+0000", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4115e-05 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:37.948032+0000", 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8290000000000002e-06 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024281+0000", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.076248809000000001 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024291+0000", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0108999999999999e-05 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.024301+0000", 2024-09-16T13:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0417e-05 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8160 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 3604480~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.088962724, 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044485497999999998, 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8160 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290710+0000", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0900000000000002e-07 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290712+0000", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9599999999999999e-06 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290718+0000", 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0909999999999998e-06 2024-09-16T13:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290722+0000", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7620000000000001e-06 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334806+0000", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044084284000000001 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.334891+0000", 2024-09-16T13:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4877999999999996e-05 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335195+0000", 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030411400000000002 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8161 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 4521984~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.08894692, 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044813883999999998, 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8161 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290725+0000", 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290726+0000", 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5600000000000001e-07 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290727+0000", 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6300000000000001e-06 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290733+0000", 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9340000000000003e-06 2024-09-16T13:30:27.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290737+0000", 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7799999999999998e-06 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335222+0000", 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044484741000000001 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335286+0000", 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3868999999999994e-05 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335539+0000", 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025357400000000001 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.414 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8162 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 5439488~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.088931359, 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046753788999999997, 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8162 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290741+0000", 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290743+0000", 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.841e-06 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290750+0000", 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2300000000000002e-06 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290755+0000", 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2139999999999998e-06 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335560+0000", 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044805734 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.335636+0000", 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6038999999999995e-05 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337495+0000", 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001858338 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8163 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 6356992~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.08891364500001, 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047813848999999999, 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8163 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290759+0000", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290761+0000", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.826e-06 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290776+0000", 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260999999999998e-05 2024-09-16T13:30:27.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290780+0000", 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8940000000000003e-06 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337536+0000", 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046755756000000002 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.337593+0000", 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7671999999999999e-05 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338572+0000", 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00097909699999999995 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8164 4.3 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:9 [sparse-read 7274496~65536] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.088887062, 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048229055, 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8164 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:27.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290785+0000", 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290786+0000", 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290788+0000", 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.216e-06 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290795+0000", 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.290798+0000", 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4300000000000002e-06 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338594+0000", 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047795211999999997 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.338674+0000", 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0857000000000002e-05 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.339014+0000", 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033983800000000002 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8657 4.b 4:d1e96943:::rbd_data.11a91a08d9ca.0000000000000090:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 108.247333149, 2024-09-16T13:30:27.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.04697001, 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8657 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132339+0000", 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132340+0000", 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3939999999999999e-06 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132348+0000", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6000000000000001e-06 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132364+0000", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132369+0000", 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.626e-06 2024-09-16T13:30:27.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132413+0000", 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4180000000000001e-05 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.132504+0000", 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0427999999999995e-05 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.134623+0000", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021194 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179232+0000", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044608385 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179240+0000", 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7479999999999999e-06 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179280+0000", 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9756999999999999e-05 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:39.179309+0000", 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8887999999999999e-05 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:21330 4.b 4:d45a83a2:::rbd_data.11a98415fd69.0000000000000088:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 105.049893578, 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.052979009000000001, 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 21330 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329779+0000", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329780+0000", 2024-09-16T13:30:27.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.341e-06 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329785+0000", 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4759999999999997e-06 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329797+0000", 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1701e-05 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329802+0000", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1279999999999999e-06 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329845+0000", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2762000000000002e-05 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.329967+0000", 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122034 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.332738+0000", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027711699999999999 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382647+0000", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049909041000000001 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382682+0000", 2024-09-16T13:30:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4513e-05 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382718+0000", 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6328000000000002e-05 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:42.382758+0000", 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9514999999999997e-05 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:38043 4.3 e59/32 ondisk, result = 0)", 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.475016972000006, 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094827540000000002, 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904655+0000", 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904657+0000", 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904664+0000", 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1720000000000001e-06 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904688+0000", 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3781000000000002e-05 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.904715+0000", 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7185000000000001e-05 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999461+0000", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094745761999999997 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999472+0000", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1061e-05 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:59.999483+0000", 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0912e-05 2024-09-16T13:30:27.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52715 4.1e 4:7a935d1f:::rbd_data.11a9fc41ce3c.00000000000001b7:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.127727014000001, 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67040195400000002, 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52715 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251945+0000", 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251946+0000", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.201e-06 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251956+0000", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4979999999999995e-06 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251968+0000", 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2345e-05 2024-09-16T13:30:27.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.251973+0000", 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4510000000000002e-06 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252021+0000", 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8171999999999999e-05 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:27.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252029+0000", 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920568+0000", 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66853896800000001 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920723+0000", 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155616 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.920860+0000", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013631299999999999 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921912+0000", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010524130000000001 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922299+0000", 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038738899999999997 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922328+0000", 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8297999999999999e-05 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922347+0000", 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9434999999999999e-05 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52718 4.1e 4:7e1f2641:::rbd_data.11a9fc41ce3c.00000000000001ba:head [set-alloc-hint object_size 524288 write_size 524288,write 131072~5097] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.127447118000006, 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.67034943999999996, 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52718 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252225+0000", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252226+0000", 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9999999999999997e-07 2024-09-16T13:30:27.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252238+0000", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2096000000000001e-05 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252248+0000", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252253+0000", 2024-09-16T13:30:27.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.198e-06 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252289+0000", 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6040000000000001e-05 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.252293+0000", 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2749999999999997e-06 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921416+0000", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.66912285100000002 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921504+0000", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7771999999999999e-05 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.921614+0000", 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110344 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922070+0000", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045565699999999999 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922528+0000", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045780499999999998 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922554+0000", 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6058999999999999e-05 2024-09-16T13:30:27.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.922575+0000", 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0616999999999999e-05 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57420 4.3 4:c686d9d7:::rbd_data.11a93cd4aec8.0000000000000009:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 73.695755402000003, 2024-09-16T13:30:27.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.049661776999999997, 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57420 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:27.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683917+0000", 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683918+0000", 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2900000000000003e-07 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683922+0000", 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0810000000000004e-06 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683932+0000", 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0246e-05 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683936+0000", 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0709999999999996e-06 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.683973+0000", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7416999999999998e-05 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.684024+0000", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0609999999999998e-05 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.686784+0000", 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027601679999999999 2024-09-16T13:30:27.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733437+0000", 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046652698999999999 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733480+0000", 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3350000000000003e-05 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733536+0000", 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5846e-05 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.733579+0000", 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2559999999999999e-05 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79913 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 7897160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:27.465 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.10168402, 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045474754999999999, 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79913 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:27.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277988+0000", 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.277990+0000", 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1119999999999999e-06 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281742+0000", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037515930000000001 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281789+0000", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7559999999999999e-05 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281800+0000", 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0107e-05 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281855+0000", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5306999999999997e-05 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.281974+0000", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011958199999999999 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282603+0000", 2024-09-16T13:30:27.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062825000000000001 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040501596000000001 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323371+0000", 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026636100000000001 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323421+0000", 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0739999999999999e-05 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323463+0000", 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1547e-05 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:79938 4.6 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.032954318999998, 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057169592999999998, 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 79938 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346718+0000", 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346720+0000", 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346728+0000", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7989999999999994e-06 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346745+0000", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6949e-05 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346752+0000", 2024-09-16T13:30:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346805+0000", 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2698000000000003e-05 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.346894+0000", 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9483999999999999e-05 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.349925+0000", 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030303550000000002 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403812+0000", 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053886983999999999 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403823+0000", 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1005000000000001e-05 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403861+0000", 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8627000000000001e-05 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403887+0000", 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6347e-05 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:79938 4.6 e82/32 ondisk, result = 0)", 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.022089579999999, 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046233403999999999, 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357583+0000", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357584+0000", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0189999999999999e-06 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357588+0000", 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:30:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357602+0000", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.347e-05 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.357608+0000", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0109999999999997e-06 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403738+0000", 2024-09-16T13:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046130506000000002 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403747+0000", 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7120000000000006e-06 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.403816+0000", 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8901999999999999e-05 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100190 4.1e 4:7a9178f5:::rbd_data.11a97e9c0852.0000000000000012:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.578005787000002, 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048355217999999998, 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100190 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801666+0000", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801667+0000", 2024-09-16T13:30:27.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801670+0000", 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801678+0000", 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3119999999999996e-06 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801681+0000", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801710+0000", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9496999999999999e-05 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.801752+0000", 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1288000000000002e-05 2024-09-16T13:30:27.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.804108+0000", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023560870000000002 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849951+0000", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045842898999999999 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849959+0000", 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5180000000000004e-06 2024-09-16T13:30:27.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.849995+0000", 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6241000000000001e-05 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.850022+0000", 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6336999999999999e-05 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.487 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103827 4.1e 4:7b9e4072:::rbd_data.11a976e61a66.0000000000000020:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.186960354999997, 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046988757999999999, 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103827 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:27.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192712+0000", 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192713+0000", 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9000000000000005e-07 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192718+0000", 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7060000000000003e-06 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192728+0000", 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0730999999999999e-05 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192733+0000", 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6079999999999998e-06 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192774+0000", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0859999999999998e-05 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.192861+0000", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7551000000000003e-05 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.195707+0000", 2024-09-16T13:30:27.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028459779999999999 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239591+0000", 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043883414000000003 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239603+0000", 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1843e-05 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239660+0000", 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7012000000000002e-05 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.239701+0000", 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1065000000000001e-05 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:27.495 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:27.492+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:27.596 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:27.594+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:27.633 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.634 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:27.634 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:27.634 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:27.696 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:27.694+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:27.740 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.741 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:27.741 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:27.741 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:27.741 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:27.834 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:27.831+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:27.889 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:27.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.184067357, 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:27.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:30:27.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.163003449, 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:30:27.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 101.25603318899999, 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:30:27.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:30:27.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.905 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100100084000005, 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:30:27.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100084214999995, 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:30:27.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.914 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100069177999998, 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:30:27.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100054510999996, 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:30:27.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:30:27.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100039769999995, 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:30:27.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:30:27.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:30:27.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.100021951000002, 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:30:27.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099837543999996, 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:30:27.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:30:27.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099819448000005, 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.937 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:30:27.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099798602000007, 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:30:27.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099783806999994, 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.946 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:30:27.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099769199999997, 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:30:27.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:30:27.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.099754743999995, 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.959 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 60.603864307999999, 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:30:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.154594297999999, 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.964 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:30:27.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:30:27.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.512452490999998, 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:27.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:30:27.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.138021858000002, 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:30:27.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:30:27.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:27.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:30:27.981 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.126619968, 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:27.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:27.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:27.985 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:27.987 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:27.987 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:27.985+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:28.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:30:28.119 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:28.116+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:28.140 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:28.140 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:28.140 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:28.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:28.190 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:28.188+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:28.290 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:28.287+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:28.291 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:28.291 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:28.291 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:28.291 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:28.291 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:28.300 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:28.391 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:28.390+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:28.439 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.30547231, 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:30:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:30:28.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:30:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:30:28.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 83.914654396000003, 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:30:28.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:30:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 83.276979926999999, 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:30:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:30:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 83.121947890000001, 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:30:28.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:30:28.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.468 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 83.116652612999999, 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:30:28.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:30:28.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 83.116094055000005, 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:30:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:30:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.443440551999998, 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:30:28.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:30:28.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.487 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.443414306999998, 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:30:28.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.443394398999999, 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:30:28.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 74.866916395999993, 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.498 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:30:28.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:30:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:30:28.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:30:28.504 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 74.701568757999993, 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.505 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:30:28.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:30:28.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:28.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 74.685376012999996, 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:30:28.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:30:28.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.558658112000003, 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.517 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:30:28.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:30:28.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.484827973000002, 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:28.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:30:28.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:30:28.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.366086737000003, 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.529 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:30:28.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.534 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.353915047999998, 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:30:28.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.799156549000003, 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:30:28.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:30:28.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.954377890000004, 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.545 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:30:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:28.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:30:28.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.944581481, 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:30:28.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 38.001681273000003, 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:30:28.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:30:28.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:30:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:30:28.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:28.563 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:28.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:28.565 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:28.562+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:28.665 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:28.663+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:28.703 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:28.703 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:28.704 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:28.704 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:28.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:28.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:28.796 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:28.794+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:28.854 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:28.854 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:28.854 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:28.854 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:28.854 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:28.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:28.867 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:28.866+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:28.869 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:28.869 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:28.886 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:28.886 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:28.901 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:28.916 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:28.916 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:28.932 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:28.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:28.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:28.963 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:28.968 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:28.965+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:28.969 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:28.969 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:28.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:28.979 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:28.979 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:29.069 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:29.067+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2024-09-16T13:30:29.094 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.094 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:29.169 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:29.166+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:29.244 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.245 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.252 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:29.296 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:29.294+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:29.351 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.351 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.359 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:29.372 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:29.370+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:29.472 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:29.470+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:29.497 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.498 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:29.573 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:29.570+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:29.605 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.605 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.614 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:29.674 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:29.672+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:29.713 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.713 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.721 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:29.806 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:29.804+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:29.860 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:29.861 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:29.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:29.911 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:29.909+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:29.976 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:29.973+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:30.017 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.017 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:30.076 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:30.073+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:30.126 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.126 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:30.178 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:30.176+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:30.232 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.233 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:30.278 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:30.276+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:30.379 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.380 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.380 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:30.378+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:30.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:30.481 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:30.479+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:30.486 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.486 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:30.583 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:30.581+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:30.594 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.594 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:30.684 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:30.682+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:30.703 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.703 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:30.785 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:30.783+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:30.810 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.810 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:30.886 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:30.885+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:30.970 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:30.970 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:30.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:30.987 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:30.985+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:30.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:30:31.086 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:31.086 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:31.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:30:31.088 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:31.086+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:31.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:31.144 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:31.160 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:31.161 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:31.176 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:31.176 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:31.188 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:31.186+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:31.205 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:31.205 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:31.223 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:31.223 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:31.239 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:31.250 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:8465 4.7 4:e6dcd048:::rbd_data.11a91a08d9ca.0000000000000095:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 112.544780449, 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.123093491, 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8465 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:31.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702515+0000", 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702519+0000", 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4790000000000001e-06 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702527+0000", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0979999999999994e-06 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702530+0000", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702556+0000", 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5999e-05 2024-09-16T13:30:31.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.702598+0000", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1347000000000002e-05 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.707647+0000", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050496559999999996 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825471+0000", 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.11782364300000001 2024-09-16T13:30:31.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825484+0000", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3288999999999999e-05 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825544+0000", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0078000000000003e-05 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825608+0000", 2024-09-16T13:30:31.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3928000000000002e-05 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:8465 4.7 e42/32 ondisk, result = 0)", 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 112.523716541, 2024-09-16T13:30:31.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101895238, 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723579+0000", 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723580+0000", 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.15e-06 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723585+0000", 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723602+0000", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7329999999999998e-05 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.723610+0000", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4120000000000004e-06 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825452+0000", 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.101841791 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825462+0000", 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0767999999999999e-05 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:38.825474+0000", 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1534e-05 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:25662 4.7 4:e12300b4:::rbd_data.11a926bfcc53.0000000000000005:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:31.261 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 104.616746281, 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043602664999999999, 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 25662 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630549+0000", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630550+0000", 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990000000000001e-06 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630556+0000", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6030000000000004e-06 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630569+0000", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3199e-05 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630574+0000", 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2024-09-16T13:30:31.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630621+0000", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6582999999999998e-05 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.630707+0000", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5922999999999997e-05 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.632993+0000", 2024-09-16T13:30:31.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002286106 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.673990+0000", 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040996988999999998 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674040+0000", 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674103+0000", 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2800999999999997e-05 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:46.674152+0000", 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8925999999999999e-05 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72861 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 933888~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.267 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460813176000002, 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892525999999997, 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72861 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786482+0000", 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2500000000000001e-07 2024-09-16T13:30:31.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786484+0000", 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854e-06 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786490+0000", 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5740000000000003e-06 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786493+0000", 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7019999999999999e-06 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825141+0000", 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.03864737 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825212+0000", 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1532000000000006e-05 2024-09-16T13:30:31.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825375+0000", 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016216900000000001 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72862 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 958464~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460797307000007, 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039118564000000002, 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72862 2024-09-16T13:30:31.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786498+0000", 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8799999999999998e-07 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786500+0000", 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8589999999999999e-06 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786506+0000", 2024-09-16T13:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786509+0000", 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6399999999999999e-06 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825402+0000", 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038892662000000001 2024-09-16T13:30:31.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825472+0000", 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0504000000000001e-05 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825616+0000", 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000144079 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.278 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72863 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 729088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460782269999996, 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039372736999999998, 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.279 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72863 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786513+0000", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786515+0000", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786521+0000", 2024-09-16T13:30:31.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786524+0000", 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825644+0000", 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039119991 2024-09-16T13:30:31.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825783+0000", 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013850700000000001 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825886+0000", 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102996 2024-09-16T13:30:31.283 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72864 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 753664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.284 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460767602999994, 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039561502999999998, 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72864 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786528+0000", 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786530+0000", 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7999999999999999e-06 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786535+0000", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5709999999999997e-06 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786539+0000", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825915+0000", 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039376095999999999 2024-09-16T13:30:31.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.825992+0000", 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7170999999999997e-05 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826089+0000", 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6982000000000001e-05 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72865 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 958464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460752862000007, 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039751712000000002, 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72865 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786542+0000", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786543+0000", 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3099999999999999e-07 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786544+0000", 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.72e-06 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786550+0000", 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5600000000000001e-06 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786554+0000", 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.698e-06 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826118+0000", 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039564031999999999 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826192+0000", 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4338000000000006e-05 2024-09-16T13:30:31.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826294+0000", 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102033 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72866 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460735043, 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040005819999999997, 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72866 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:31.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786560+0000", 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786561+0000", 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786562+0000", 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:30:31.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786568+0000", 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786572+0000", 2024-09-16T13:30:31.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4510000000000001e-06 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826428+0000", 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039856056000000001 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826501+0000", 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3311000000000003e-05 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826566+0000", 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4919000000000006e-05 2024-09-16T13:30:31.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72943 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460550635999994, 2024-09-16T13:30:31.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040003924000000003, 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72943 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786745+0000", 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786750+0000", 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786755+0000", 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786759+0000", 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826319+0000", 2024-09-16T13:30:31.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039560505000000003 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826632+0000", 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031260899999999999 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826748+0000", 2024-09-16T13:30:31.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011645 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72944 4.2 4:466d48ea:::rbd_data.11a911a8a02a.000000000000002a:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460532540000003, 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041979666999999998, 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.313 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72944 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786763+0000", 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2599999999999998e-07 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786770+0000", 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8759999999999999e-06 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786776+0000", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9179999999999999e-06 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786779+0000", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826991+0000", 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040211447999999997 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.827116+0000", 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125387 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828742+0000", 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016260409999999999 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72945 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460511694000004, 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041790978999999999, 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.318 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72945 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786783+0000", 2024-09-16T13:30:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786784+0000", 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3000000000000002e-07 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786786+0000", 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786791+0000", 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5890000000000002e-06 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786795+0000", 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826849+0000", 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040053985 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.826932+0000", 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3670000000000004e-05 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828574+0000", 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016421949999999999 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72946 4.12 4:4886011f:::rbd_data.11a911a8a02a.000000000000002b:1d [read 892928~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.323 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460496899000006, 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041937734999999997, 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72946 2024-09-16T13:30:31.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786798+0000", 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786799+0000", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786801+0000", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786806+0000", 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4049999999999999e-06 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786809+0000", 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999998e-06 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828597+0000", 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041788068999999997 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828667+0000", 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9332000000000006e-05 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828736+0000", 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9309999999999999e-05 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.328 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72947 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460482291999995, 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042188864, 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72947 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786813+0000", 2024-09-16T13:30:31.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999998e-07 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786815+0000", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.902e-06 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786820+0000", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3279999999999996e-06 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786824+0000", 2024-09-16T13:30:31.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828759+0000", 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041935186999999999 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.828939+0000", 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017974000000000001 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829002+0000", 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:72948 4.12 4:4bbb1632:::rbd_data.11a911a8a02a.000000000000002c:1d [read 917504~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.460467836000007, 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042315773000000001, 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.333 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 72948 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786827+0000", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786828+0000", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786830+0000", 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8419999999999999e-06 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786835+0000", 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5729999999999998e-06 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.786839+0000", 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829023+0000", 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042184264999999999 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829089+0000", 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6508999999999996e-05 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:23.829143+0000", 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3641999999999999e-05 2024-09-16T13:30:31.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:79913 4.6 e82/32 4:670847e6:::rbd_data.11a9ec63c06f.0000000000000001:head v 82'257, mlcod=82'255)", 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.964577400000003, 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040446236000000003, 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:31.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282718+0000", 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.282720+0000", 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.748e-06 2024-09-16T13:30:31.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288247+0000", 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055276850000000001 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288274+0000", 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6635000000000002e-05 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288284+0000", 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7780000000000002e-06 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288328+0000", 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4118e-05 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.288355+0000", 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7628999999999999e-05 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323104+0000", 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034748344 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.323164+0000", 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299e-05 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.342 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:100148 4.12 4:4a1d9238:::rbd_data.11a97e9c0852.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 49.515307389999997, 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040585867999999997, 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 100148 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731988+0000", 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.731989+0000", 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4759999999999999e-06 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733670+0000", 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001680546 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733692+0000", 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2107999999999999e-05 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733702+0000", 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.611e-06 2024-09-16T13:30:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733786+0000", 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4210999999999997e-05 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.733962+0000", 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017615999999999999 2024-09-16T13:30:31.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.734341+0000", 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037933500000000001 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.761353+0000", 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027011901000000001 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772502+0000", 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011148590999999999 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772538+0000", 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5896000000000003e-05 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:41.772574+0000", 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6032999999999999e-05 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103860 4.4 4:248db71f:::rbd_data.11a976e61a66.0000000000000022:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.873165583000002, 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040568282999999997, 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103860 2024-09-16T13:30:31.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374130+0000", 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8299999999999996e-07 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374134+0000", 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.72e-06 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374143+0000", 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3309999999999993e-06 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374149+0000", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3199999999999999e-06 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374184+0000", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5200000000000002e-05 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.374268+0000", 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4115000000000004e-05 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.376646+0000", 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023779959999999998 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414596+0000", 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037949699000000003 2024-09-16T13:30:31.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414610+0000", 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4228e-05 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414661+0000", 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0557999999999999e-05 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.414698+0000", 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7333000000000003e-05 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:104013 4.2 4:46fb88c4:::rbd_data.11a976e61a66.00000000000000dc:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.498734949999999, 2024-09-16T13:30:31.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.057937797999999999, 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 104013 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748560+0000", 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748561+0000", 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0100000000000005e-07 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748564+0000", 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748573+0000", 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748576+0000", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7780000000000001e-06 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748603+0000", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6642999999999999e-05 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.748678+0000", 2024-09-16T13:30:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5059000000000001e-05 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5", 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.751816+0000", 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031374440000000001 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806376+0000", 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054560770000000001 2024-09-16T13:30:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806429+0000", 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2262000000000002e-05 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806470+0000", 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1207e-05 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806498+0000", 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8246e-05 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:104013 4.2 e99/32 ondisk, result = 0)", 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 44.487333059999997, 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046541580999999999, 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759962+0000", 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759963+0000", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759967+0000", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.177e-06 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759979+0000", 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2038999999999999e-05 2024-09-16T13:30:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.759986+0000", 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0800000000000002e-06 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806417+0000", 2024-09-16T13:30:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046431130000000001 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806424+0000", 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9670000000000004e-06 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.806504+0000", 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0080999999999994e-05 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.368 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:31.369 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:31.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:31.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:31.371 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:31.372 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:31.372 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:31.368+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:31.471 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:31.469+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:31.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:31.572 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:31.570+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:31.614 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:31.615 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:31.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:31.673 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:31.672+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:31.762 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:2419 4.1c 4:3d288900:::rbd_data.11a962dad130.0000000000000040:head [delete] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 117.62822344, 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095880619999999996, 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2419 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131718+0000", 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131719+0000", 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6199999999999998e-07 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131721+0000", 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131728+0000", 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7399999999999998e-06 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.131732+0000", 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.878e-06 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134397+0000", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026650749999999998 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134471+0000", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3737000000000002e-05 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.134636+0000", 2024-09-16T13:30:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016521999999999999 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.135724+0000", 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010877739999999999 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141222+0000", 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005498251 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141266+0000", 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3823999999999999e-05 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:28:34.141306+0000", 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0581999999999999e-05 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49153 4.f 4:f7a4ea9f:::rbd_data.11a9b5ab4b7f.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.237405526000003, 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098544549999999998, 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49153 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522536+0000", 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.522539+0000", 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7630000000000001e-06 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523256+0000", 2024-09-16T13:30:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071650700000000002 2024-09-16T13:30:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523279+0000", 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3923000000000001e-05 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523288+0000", 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6260000000000001e-06 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523353+0000", 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4894000000000006e-05 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523513+0000", 2024-09-16T13:30:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015971299999999999 2024-09-16T13:30:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.523966+0000", 2024-09-16T13:30:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045344299999999999 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532198+0000", 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008232184 2024-09-16T13:30:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532283+0000", 2024-09-16T13:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4696000000000004e-05 2024-09-16T13:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532345+0000", 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2113000000000007e-05 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:04.532391+0000", 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5593000000000002e-05 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.783 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:49735 4.1c 4:3b13e5fb:::rbd_header.11a9b5ab4b7f:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:31.783 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:31.783 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.599731057, 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012366633, 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 49735 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:31.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160211+0000", 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160213+0000", 2024-09-16T13:30:31.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.075e-06 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160222+0000", 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5889999999999992e-06 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160244+0000", 2024-09-16T13:30:31.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1365999999999998e-05 2024-09-16T13:30:31.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160253+0000", 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4439999999999997e-06 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160314+0000", 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0278000000000001e-05 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.160322+0000", 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5169999999999999e-06 2024-09-16T13:30:31.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162293+0000", 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019712520000000002 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162352+0000", 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8646000000000001e-05 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.162650+0000", 2024-09-16T13:30:31.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029790200000000001 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.164257+0000", 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001606679 2024-09-16T13:30:31.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172480+0000", 2024-09-16T13:30:31.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082238799999999994 2024-09-16T13:30:31.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172533+0000", 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2574000000000002e-05 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.172577+0000", 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4431000000000002e-05 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:51133 4.4 e65/32 4:23bd15ef:::rbd_data.11a9b5ab4b7f.0000000000000002:head v 65'103, mlcod=62'101)", 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.444699020000002, 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014745652999999999, 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315243+0000", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315244+0000", 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.24e-06 2024-09-16T13:30:31.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315816+0000", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057173599999999999 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315837+0000", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169e-05 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315845+0000", 2024-09-16T13:30:31.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0390000000000005e-06 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315911+0000", 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6402000000000006e-05 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.315930+0000", 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8757e-05 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329959+0000", 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014028850000000001 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329988+0000", 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999999e-05 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51139 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:31.801 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.439403743, 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013441130000000001, 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51139 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:31.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320538+0000", 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.320540+0000", 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321058+0000", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00051757599999999995 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321077+0000", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9321e-05 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321085+0000", 2024-09-16T13:30:31.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8159999999999994e-06 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321134+0000", 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9543000000000003e-05 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321268+0000", 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013367199999999999 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321522+0000", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253742 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329998+0000", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084759880000000003 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333918+0000", 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039206559999999998 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333953+0000", 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5064999999999997e-05 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.333979+0000", 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5760000000000001e-05 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.807 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51140 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:head [truncate 606208] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.438845185000005, 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013138705000000001, 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51140 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321097+0000", 2024-09-16T13:30:31.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7700000000000004e-07 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321100+0000", 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321107+0000", 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5899999999999996e-06 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.321111+0000", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328840+0000", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077291679999999998 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.328921+0000", 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1408000000000004e-05 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.329079+0000", 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015773999999999999 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.330228+0000", 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011491120000000001 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334186+0000", 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039576749999999999 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334214+0000", 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8068999999999998e-05 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:05.334235+0000", 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.137e-05 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53301 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 565248~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 80.766191681999999, 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095014890000000001, 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.814 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53301 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993750+0000", 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993751+0000", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0299999999999999e-07 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993753+0000", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4739999999999999e-06 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993760+0000", 2024-09-16T13:30:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.596e-06 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993770+0000", 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0540000000000001e-05 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003074+0000", 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093039219999999992 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003152+0000", 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7805999999999993e-05 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003252+0000", 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9648000000000005e-05 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53310 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 909312~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 80.766165436999998, 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096506820000000007, 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53310 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:31.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993776+0000", 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993777+0000", 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2e-07 2024-09-16T13:30:31.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993779+0000", 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2539999999999999e-06 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993785+0000", 2024-09-16T13:30:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3280000000000002e-06 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993789+0000", 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6569999999999999e-06 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003305+0000", 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0095156259999999993 2024-09-16T13:30:31.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003361+0000", 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6189e-05 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003427+0000", 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6208e-05 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:53315 4.1c 4:3c7b2831:::rbd_data.11a9b5ab4b7f.0000000000000004:16 [read 1024000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 80.766145528999999, 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097690559999999999, 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.824 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 53315 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993796+0000", 2024-09-16T13:30:31.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993797+0000", 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993799+0000", 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9580000000000001e-06 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993804+0000", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.993808+0000", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.427e-06 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003450+0000", 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096419590000000003 2024-09-16T13:30:31.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003506+0000", 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6530999999999998e-05 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:11.003565+0000", 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9014000000000001e-05 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57335 4.1c 4:3e8028d3:::rbd_data.11a93cd4aec8.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 2621440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.189667525999994, 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010438965999999999, 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.829 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57335 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570274+0000", 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570277+0000", 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.26e-06 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570580+0000", 2024-09-16T13:30:31.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030324200000000001 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570601+0000", 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1497000000000001e-05 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570610+0000", 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.473e-06 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570663+0000", 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2941e-05 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570810+0000", 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147362 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.570973+0000", 2024-09-16T13:30:31.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016307299999999999 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580232+0000", 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.009259125 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580623+0000", 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039028000000000001 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580675+0000", 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.295e-05 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.580713+0000", 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7762999999999997e-05 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:57422 4.1c 4:39649c9f:::rbd_data.11a93cd4aec8.000000000000000a:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.024319887999994, 2024-09-16T13:30:31.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.094201883, 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 57422 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735622+0000", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735623+0000", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735630+0000", 2024-09-16T13:30:31.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9630000000000001e-06 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735646+0000", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5798e-05 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735653+0000", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4670000000000001e-06 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735698+0000", 2024-09-16T13:30:31.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.735751+0000", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2562999999999998e-05 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.739727+0000", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039761379999999997 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829714+0000", 2024-09-16T13:30:31.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.089986848999999994 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829754+0000", 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9706999999999998e-05 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829797+0000", 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3779999999999998e-05 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829824+0000", 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6503e-05 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:57422 4.1c e73/32 ondisk, result = 0)", 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 78.008127142999996, 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078015307000000006, 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751815+0000", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751819+0000", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751832+0000", 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.275e-05 2024-09-16T13:30:31.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.751837+0000", 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5319999999999997e-06 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829741+0000", 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077903320999999998 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829749+0000", 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7239999999999998e-06 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:13.829830+0000", 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0493000000000003e-05 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:90396 4.f 4:f40d60bc:::rbd_data.11a93542739e.0000000000000008:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~16384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:31.845 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.881409241999997, 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010982578999999999, 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 90396 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:31.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878533+0000", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.878535+0000", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3920000000000001e-06 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879270+0000", 2024-09-16T13:30:31.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073472200000000004 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879288+0000", 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7906000000000001e-05 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879296+0000", 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9670000000000001e-06 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879339+0000", 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3415999999999999e-05 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879451+0000", 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011168800000000001 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.879686+0000", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023510499999999999 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889318+0000", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096326220000000004 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889438+0000", 2024-09-16T13:30:31.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119726 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889472+0000", 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4258000000000003e-05 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.889515+0000", 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2777000000000001e-05 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:90826 4.d e88/32 4:b2754109:::rbd_data.11a93542739e.000000000000000d:head v 88'330, mlcod=88'328)", 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.807579103000002, 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098732009999999999, 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952363+0000", 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.952364+0000", 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.291e-06 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954264+0000", 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001900316 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954287+0000", 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.283e-05 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954294+0000", 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954339+0000", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5309999999999998e-05 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.954358+0000", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8799000000000001e-05 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962196+0000", 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078386629999999992 2024-09-16T13:30:31.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:33.962236+0000", 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9585000000000001e-05 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92719 4.d e88/32 4:b41d5a12:::rbd_data.11a93542739e.0000000000000002:head v 88'333, mlcod=88'331)", 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.688837866999997, 2024-09-16T13:30:31.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011698853, 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071104+0000", 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.071105+0000", 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4729999999999999e-06 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072927+0000", 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018215950000000001 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072947+0000", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9840999999999999e-05 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072955+0000", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.337e-06 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.072995+0000", 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9579999999999997e-05 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.073016+0000", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1650000000000001e-05 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082746+0000", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0097298049999999994 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.082803+0000", 2024-09-16T13:30:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6572e-05 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:92728 4.4 e88/32 4:205a7f3e:::rbd_data.11a93542739e.0000000000000006:head v 88'222, mlcod=88'220)", 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.676666177999998, 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011854828, 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083276+0000", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083277+0000", 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.158e-06 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083952+0000", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067497499999999997 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083963+0000", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1457999999999999e-05 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.083971+0000", 2024-09-16T13:30:31.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084015+0000", 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4379000000000003e-05 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.084041+0000", 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5505999999999999e-05 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095074+0000", 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011033695 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:34.095130+0000", 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6240999999999999e-05 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.4521.0:101413 4.19 e96/32 4:9c88894a:::rbd_data.11a983f88957.0000000000000001:head v 96'262, mlcod=95'259)", 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:31.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 49.121907679000003, 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019522174, 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638034+0000", 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638036+0000", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.404e-06 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638778+0000", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074217100000000002 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638795+0000", 2024-09-16T13:30:31.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7159999999999998e-05 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638804+0000", 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2499999999999995e-06 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638860+0000", 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5597999999999999e-05 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.638886+0000", 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657513+0000", 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018626797 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:42.657556+0000", 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3711000000000002e-05 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103894 4.1c 4:3c068e0d:::rbd_data.11a976e61a66.000000000000015d:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.277129019999997, 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051185438, 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.870 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103894 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482813+0000", 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482815+0000", 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482824+0000", 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482844+0000", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9409999999999999e-05 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482852+0000", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0139999999999998e-06 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482894+0000", 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2555000000000002e-05 2024-09-16T13:30:31.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.482986+0000", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299000000000005e-05 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.485956+0000", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029693520000000002 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533881+0000", 2024-09-16T13:30:31.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047925574999999998 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533894+0000", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2428e-05 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533953+0000", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9506000000000003e-05 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533998+0000", 2024-09-16T13:30:31.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4971e-05 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103894 4.1c e99/32 ondisk, result = 0)", 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.267332611, 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041274938999999997, 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492609+0000", 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492610+0000", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492614+0000", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492626+0000", 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2145e-05 2024-09-16T13:30:31.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.492631+0000", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3879999999999999e-06 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533862+0000", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041231300999999998 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533874+0000", 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1681e-05 2024-09-16T13:30:31.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:46.533884+0000", 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9699999999999994e-06 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:106307 4.f 4:f1f03fa0:::rbd_header.11a98b56945c:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.324432403000003, 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093699560000000005, 2024-09-16T13:30:31.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 106307 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435509+0000", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435512+0000", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435522+0000", 2024-09-16T13:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9750000000000002e-06 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435546+0000", 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4059000000000001e-05 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435556+0000", 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0402e-05 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435634+0000", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7350999999999999e-05 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.435641+0000", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8790000000000003e-06 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438582+0000", 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002940181 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.438692+0000", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110697 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.439086+0000", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039406700000000001 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.441736+0000", 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026493269999999999 2024-09-16T13:30:31.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444780+0000", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030447429999999999 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444837+0000", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6314000000000003e-05 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:50.444879+0000", 2024-09-16T13:30:31.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2611999999999998e-05 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:31.887 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:31.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:30:31.888 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:30:31.888 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T13:30:31.888 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:30:31.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:31.890 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:31.887+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:30:31.990 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:31.988+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:32.046 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:32.047 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:32.047 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2024-09-16T13:30:32.047 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:32.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:32.090 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:32.088+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:32.191 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:32.189+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:32.198 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:32.198 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2024-09-16T13:30:32.199 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2024-09-16T13:30:32.199 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2024-09-16T13:30:32.199 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:32.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:32.223 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:32.223 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:32.223 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 144, 2024-09-16T13:30:32.223 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "0cf46de4-f58b-4e68-a8b9-534bb33f8d72", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T13:27:50.121006+0000", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T13:30:31.924491+0000", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2024-09-16T13:28:03.264780+0000", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2024-09-16T13:29:06.165279+0000", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5804032, 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "nodeep-scrub", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2024-09-16T13:30:32.224 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 5, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 4, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2024-09-16T13:30:32.225 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".mgr", 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:03.867975+0000", 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:30:32.226 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:30:32.227 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "18", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2024-09-16T13:30:32.228 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:30:32.229 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:30:32.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_max": 32, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": {} 2024-09-16T13:30:32.231 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "rbd", 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:08.815950+0000", 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:30:32.232 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 8, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 8, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 8, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 8, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 8, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:30:32.233 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "27", 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:30:32.234 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [ 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": 3, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": 3, 2024-09-16T13:30:32.235 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.236 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:30:32.237 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "cache", 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:16.900065+0000", 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8201, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:30:32.238 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 4, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 4, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 4, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 4, 2024-09-16T13:30:32.239 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 4, 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "31", 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "27", 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "27", 2024-09-16T13:30:32.240 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "27", 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 3, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 25, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": 2, 2024-09-16T13:30:32.241 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "writeback", 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 250, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:30:32.242 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bloom", 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "false_positive_probability": 0.050000000000000003, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "target_size": 0, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "seed": 0 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 60, 2024-09-16T13:30:32.243 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 8, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [ 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.244 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: 1000000 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "pool_client.0", 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2024-09-16T13:28:25.583589+0000", 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 2, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 1, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:30:32.246 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2024-09-16T13:30:32.247 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "144", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 90, 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 144, 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2024-09-16T13:30:32.248 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2024-09-16T13:30:32.249 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2024-09-16T13:30:32.250 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "daa6f01d-c106-4b1d-8063-1852db16dec3", 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.251 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2024-09-16T13:30:32.253 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2024-09-16T13:30:32.255 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33372 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6817/33372", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6819/33372", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6823/33372", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6821/33372", 2024-09-16T13:30:32.256 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "70bfb231-8371-4267-83d7-8c7bdf4d122b", 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.257 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33371 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6801/33371", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6803/33371", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6807/33371", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6805/33371", 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a4447a8c-bda5-4387-bc24-77c6cc28374b", 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.6766510009765625, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.263 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2024-09-16T13:30:32.265 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33387 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6809/33387", 2024-09-16T13:30:32.268 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6811/33387", 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6815/33387", 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6813/33387", 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "50031c12-2464-4373-b82a-2d4da1dae766", 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.269 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 70, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.270 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.271 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.272 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33388 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6825/33388", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6827/33388", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6831/33388", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6829/33388", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "24b2d993-ea4d-406e-adbd-e06d278de26d", 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 32, 2024-09-16T13:30:32.276 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6808", 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6809", 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6811", 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6813", 2024-09-16T13:30:32.278 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6821", 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.279 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6824", 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6815", 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6818", 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33329 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6809/33329", 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6813/33329", 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6824/33329", 2024-09-16T13:30:32.281 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6818/33329", 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.282 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "d68602fa-7b2b-43fc-96ba-f807ca71872e", 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0, 2024-09-16T13:30:32.283 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 96, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.284 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6817", 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6820", 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6823", 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6827", 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6830", 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6831", 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6828", 2024-09-16T13:30:32.288 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6829", 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33357 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6820/33357", 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6827/33357", 2024-09-16T13:30:32.289 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6831/33357", 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6829/33357", 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "5f02e46d-718f-454b-99c5-245ded452730", 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.290 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 14, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 66, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6800", 2024-09-16T13:30:32.291 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6801", 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.292 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6802", 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6803", 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.293 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6806", 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6807", 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.294 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6804", 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6805", 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33330 2024-09-16T13:30:32.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6801/33330", 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6803/33330", 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6807/33330", 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6805/33330", 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:30:32.296 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "ea09d724-4028-4cd4-b5b5-9d0122dd5495", 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 15, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 106, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6810", 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6812", 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6814", 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6816", 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6825", 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6826", 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T13:30:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6819", 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.163:6822", 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 33331 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.163:6812/33331", 2024-09-16T13:30:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.163:6816/33331", 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.163:6826/33331", 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.163:6822/33331", 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.302 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:28:01.464981+0000", 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.067355+0000", 2024-09-16T13:30:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.138713+0000", 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2024-09-16T13:30:32.304 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.125791+0000", 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.305 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.187216+0000", 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.306 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.205813+0000", 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.086803+0000", 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138320759226367, 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2024-09-16T13:27:58.207997+0000", 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.308 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.5", 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.d", 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.309 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.2", 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2024-09-16T13:30:32.310 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": {}, 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "range_blocklist": {}, 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2024-09-16T13:30:32.311 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 5, 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 9 2024-09-16T13:30:32.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 5, 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.314 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:32.317 INFO:tasks.thrashosds.thrasher:j is {'epoch': 144, 'fsid': '0cf46de4-f58b-4e68-a8b9-534bb33f8d72', 'created': '2024-09-16T13:27:50.121006+0000', 'modified': '2024-09-16T13:30:31.924491+0000', 'last_up_change': '2024-09-16T13:28:03.264780+0000', 'last_in_change': '2024-09-16T13:29:06.165279+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:28:03.867975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:28:08.815950+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-09-16T13:28:16.900065+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '27', 'last_force_op_resend_prenautilus': '27', 'last_force_op_resend_preluminous': '27', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 25, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:28:25.583589+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 90, 'snap_epoch': 144, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'daa6f01d-c106-4b1d-8063-1852db16dec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 33372}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 33372}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 33372}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 33372}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 33372}]}, 'public_addr': '172.21.15.76:6817/33372', 'cluster_addr': '172.21.15.76:6819/33372', 'heartbeat_back_addr': '172.21.15.76:6823/33372', 'heartbeat_front_addr': '172.21.15.76:6821/33372', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '70bfb231-8371-4267-83d7-8c7bdf4d122b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6800', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6801', 'nonce': 33371}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 33371}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 33371}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 33371}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 33371}]}, 'public_addr': '172.21.15.76:6801/33371', 'cluster_addr': '172.21.15.76:6803/33371', 'heartbeat_back_addr': '172.21.15.76:6807/33371', 'heartbeat_front_addr': '172.21.15.76:6805/33371', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a4447a8c-bda5-4387-bc24-77c6cc28374b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6766510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 33387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 33387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 33387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 33387}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 33387}]}, 'public_addr': '172.21.15.76:6809/33387', 'cluster_addr': '172.21.15.76:6811/33387', 'heartbeat_back_addr': '172.21.15.76:6815/33387', 'heartbeat_front_addr': '172.21.15.76:6813/33387', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '50031c12-2464-4373-b82a-2d4da1dae766', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 70, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 33388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 33388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 33388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 33388}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 33388}]}, 'public_addr': '172.21.15.76:6825/33388', 'cluster_addr': '172.21.15.76:6827/33388', 'heartbeat_back_addr': '172.21.15.76:6831/33388', 'heartbeat_front_addr': '172.21.15.76:6829/33388', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '24b2d993-ea4d-406e-adbd-e06d278de26d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6808', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 33329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6811', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 33329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6821', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6824', 'nonce': 33329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6815', 'nonce': 33329}, {'type': 'v1', 'addr': '172.21.15.163:6818', 'nonce': 33329}]}, 'public_addr': '172.21.15.163:6809/33329', 'cluster_addr': '172.21.15.163:6813/33329', 'heartbeat_back_addr': '172.21.15.163:6824/33329', 'heartbeat_front_addr': '172.21.15.163:6818/33329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd68602fa-7b2b-43fc-96ba-f807ca71872e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6817', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6820', 'nonce': 33357}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6823', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6827', 'nonce': 33357}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6830', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6831', 'nonce': 33357}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6828', 'nonce': 33357}, {'type': 'v1', 'addr': '172.21.15.163:6829', 'nonce': 33357}]}, 'public_addr': '172.21.15.163:6820/33357', 'cluster_addr': '172.21.15.163:6827/33357', 'heartbeat_back_addr': '172.21.15.163:6831/33357', 'heartbeat_front_addr': '172.21.15.163:6829/33357', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5f02e46d-718f-454b-99c5-245ded452730', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6800', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 33330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6802', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 33330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6806', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 33330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6804', 'nonce': 33330}, {'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 33330}]}, 'public_addr': '172.21.15.163:6801/33330', 'cluster_addr': '172.21.15.163:6803/33330', 'heartbeat_back_addr': '172.21.15.163:6807/33330', 'heartbeat_front_addr': '172.21.15.163:6805/33330', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ea09d724-4028-4cd4-b5b5-9d0122dd5495', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6810', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 33331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6814', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 33331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6825', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6826', 'nonce': 33331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6819', 'nonce': 33331}, {'type': 'v1', 'addr': '172.21.15.163:6822', 'nonce': 33331}]}, 'public_addr': '172.21.15.163:6812/33331', 'cluster_addr': '172.21.15.163:6816/33331', 'heartbeat_back_addr': '172.21.15.163:6826/33331', 'heartbeat_front_addr': '172.21.15.163:6822/33331', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:28:01.464981+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.067355+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.138713+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.125791+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.187216+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.205813+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.086803+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:27:58.207997+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [6, 1]}, {'pgid': '4.d', 'osds': [5, 0]}], 'pg_upmap_items': [{'pgid': '2.2', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 5, 'length': 9}]}], 'new_removed_snaps': [{'pool': 4, 'snaps': [{'begin': 5, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:30:32.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:32.318 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:32.314+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:32.417 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:30:32.417 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2024-09-16T13:30:32.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2024-09-16T13:30:32.417 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2024-09-16T13:30:32.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51673 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 417792~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171836768000006, 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7923316099999997, 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51673 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242249+0000", 2024-09-16T13:30:32.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242250+0000", 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6200000000000006e-07 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242253+0000", 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7529999999999999e-06 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242264+0000", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1019000000000001e-05 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242270+0000", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.344e-06 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242320+0000", 2024-09-16T13:30:32.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0501000000000003e-05 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242331+0000", 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0431000000000001e-05 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179974+0000", 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93764299200000001 2024-09-16T13:30:32.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.179983+0000", 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8619999999999998e-06 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034320+0000", 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.854336094 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034453+0000", 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013379099999999999 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034580+0000", 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000126861 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51688 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171774295000006, 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7924712700000001, 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51688 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242311+0000", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242314+0000", 2024-09-16T13:30:32.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3640000000000002e-06 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242322+0000", 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4139999999999994e-06 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242326+0000", 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6899999999999998e-06 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242403+0000", 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7461999999999998e-05 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242413+0000", 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3060000000000003e-06 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180006+0000", 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93759343299999998 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180013+0000", 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4019999999999999e-06 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034600+0000", 2024-09-16T13:30:32.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8545878089999999 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034671+0000", 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0988000000000005e-05 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034782+0000", 2024-09-16T13:30:32.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110955 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51692 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 516096~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171729642000003, 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.792754634, 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51692 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:32.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242356+0000", 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242358+0000", 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:30:32.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242364+0000", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.789e-06 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242367+0000", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.45e-06 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242551+0000", 2024-09-16T13:30:32.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018391499999999999 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242557+0000", 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.728e-06 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180034+0000", 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93747769199999997 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180041+0000", 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034829+0000", 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8547878469999999 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.034929+0000", 2024-09-16T13:30:32.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100208 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035110+0000", 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018156300000000001 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51698 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 532480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171656541000004, 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7929635240000001, 2024-09-16T13:30:32.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51698 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.441 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242429+0000", 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242432+0000", 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6419999999999999e-06 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242439+0000", 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3370000000000002e-06 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242443+0000", 2024-09-16T13:30:32.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0330000000000002e-06 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242712+0000", 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026844899999999998 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242717+0000", 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.972e-06 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180067+0000", 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735016900000001 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180073+0000", 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3339999999999998e-06 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035137+0000", 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855063267 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035225+0000", 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8117999999999999e-05 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035392+0000", 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016769300000000001 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51699 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 548864~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171638035000001, 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7932572540000002, 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.449 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51699 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242447+0000", 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242448+0000", 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5699999999999998e-07 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242450+0000", 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9740000000000001e-06 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242456+0000", 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.066e-06 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242459+0000", 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6179999999999999e-06 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242733+0000", 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273594 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242736+0000", 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3629999999999998e-06 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180094+0000", 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93735806300000002 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180101+0000", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035419+0000", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855318142 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035505+0000", 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6458000000000005e-05 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035705+0000", 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019936500000000001 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51715 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 483328~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171464083000004, 2024-09-16T13:30:32.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7934491399999999, 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51715 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242621+0000", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242622+0000", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.75e-07 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242625+0000", 2024-09-16T13:30:32.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9550000000000001e-06 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242632+0000", 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6560000000000003e-06 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242635+0000", 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2179999999999999e-06 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242966+0000", 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000331484 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242970+0000", 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6169999999999998e-06 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180122+0000", 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93715201199999998 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180128+0000", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3300000000000004e-06 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035729+0000", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.855601016 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.035829+0000", 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9710999999999995e-05 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036070+0000", 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000241466 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51717 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 466944~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.171425529000004, 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937175160000001, 2024-09-16T13:30:32.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51717 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242660+0000", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2499999999999999e-07 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242661+0000", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4479999999999999e-06 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242665+0000", 2024-09-16T13:30:32.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0400000000000001e-06 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.242666+0000", 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.978e-06 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243122+0000", 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000455957 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243128+0000", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180149+0000", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93702100099999996 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180155+0000", 2024-09-16T13:30:32.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8889999999999999e-06 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036139+0000", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8559842610000001 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036217+0000", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8399000000000006e-05 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036377+0000", 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016001500000000001 2024-09-16T13:30:32.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51759 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 663552~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.170466834999999, 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930493539999999, 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.467 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51759 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243619+0000", 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3799999999999997e-07 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243623+0000", 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243631+0000", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2670000000000006e-06 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243636+0000", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6990000000000002e-06 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243681+0000", 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5188000000000002e-05 2024-09-16T13:30:32.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243689+0000", 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5909999999999998e-06 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180175+0000", 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.936486029 2024-09-16T13:30:32.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180178+0000", 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.58e-06 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036415+0000", 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8562363039999998 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036508+0000", 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3511999999999995e-05 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036668+0000", 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159782 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.473 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51763 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 679936~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.170409038000003, 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7933561500000001, 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.474 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51763 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243676+0000", 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243677+0000", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243679+0000", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4820000000000001e-06 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243687+0000", 2024-09-16T13:30:32.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1780000000000003e-06 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243692+0000", 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0609999999999997e-06 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243726+0000", 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4378999999999997e-05 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243732+0000", 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180189+0000", 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93645780700000003 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180193+0000", 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1289999999999998e-06 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036697+0000", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.856503384 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.036804+0000", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107415 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037032+0000", 2024-09-16T13:30:32.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022816700000000001 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51792 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 434176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.170315161999994, 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7935202229999998, 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.480 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51792 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243770+0000", 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243771+0000", 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3499999999999996e-07 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243774+0000", 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.96e-06 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243784+0000", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.025e-05 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243787+0000", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.309e-06 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243868+0000", 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0914000000000005e-05 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243873+0000", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8590000000000004e-06 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180212+0000", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93633926099999998 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180226+0000", 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3832000000000001e-05 2024-09-16T13:30:32.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037057+0000", 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8568311309999999 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037143+0000", 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6106999999999994e-05 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037290+0000", 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014696500000000001 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51793 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 450560~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.170287173000006, 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7937701430000002, 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.486 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51793 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243798+0000", 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243799+0000", 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243801+0000", 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243805+0000", 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2939999999999999e-06 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243808+0000", 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.278e-06 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243884+0000", 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6736999999999994e-05 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.243888+0000", 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1080000000000001e-06 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180266+0000", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93637833999999998 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180272+0000", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3110000000000002e-06 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037315+0000", 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.857042321 2024-09-16T13:30:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037407+0000", 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2509000000000005e-05 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037568+0000", 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016124299999999999 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51829 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 630784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.169013147000001, 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7927876830000002, 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51829 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245072+0000", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245073+0000", 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0299999999999998e-07 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245076+0000", 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245085+0000", 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245090+0000", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7790000000000003e-06 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245123+0000", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2557e-05 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245139+0000", 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6047000000000001e-05 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180297+0000", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93515799399999999 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180303+0000", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7630000000000003e-06 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037597+0000", 2024-09-16T13:30:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8572939079999999 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037691+0000", 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3636999999999998e-05 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037860+0000", 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016888499999999999 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:51830 4.0 4:06d7495c:::rbd_data.11a9441c6517.0000000000000003:15 [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.168989901000003, 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7930066409999998, 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 51830 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245095+0000", 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245096+0000", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245098+0000", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.951e-06 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245104+0000", 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1929999999999998e-06 2024-09-16T13:30:32.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245108+0000", 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8120000000000001e-06 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245170+0000", 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2506000000000001e-05 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:06.245176+0000", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4990000000000002e-06 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180324+0000", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.93514778499999995 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:07.180330+0000", 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:30:32.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037884+0000", 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8575536860000001 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.037970+0000", 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6113999999999996e-05 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.038102+0000", 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013218799999999999 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:52781 4.0 4:06c0073d:::rbd_data.11a9fc41ce3c.0000000000000065:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.441643752000004, 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.94662011599999996, 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.505 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 52781 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972442+0000", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8700000000000004e-07 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972470+0000", 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7161999999999998e-05 2024-09-16T13:30:32.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972482+0000", 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1867e-05 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972486+0000", 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4299999999999999e-06 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972532+0000", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6207e-05 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:09.972538+0000", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8459999999999996e-06 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916643+0000", 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.944105054 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.916762+0000", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118966 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.917043+0000", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028083300000000003 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.918532+0000", 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014893479999999999 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919007+0000", 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047502500000000001 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919040+0000", 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2471999999999997e-05 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:10.919062+0000", 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2019e-05 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75175 4.11 4:8ecca89b:::rbd_data.11a9cafed80f.00000000000000ac:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.137015849999997, 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073468413999999996, 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75175 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277069+0000", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277071+0000", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.153e-06 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277077+0000", 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4559999999999997e-06 2024-09-16T13:30:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277090+0000", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2527999999999999e-05 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277095+0000", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1730000000000003e-06 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277143+0000", 2024-09-16T13:30:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7716000000000002e-05 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.277297+0000", 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000154136 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.280859+0000", 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623959999999998 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350395+0000", 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069536100000000003 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350444+0000", 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8368999999999997e-05 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350505+0000", 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1761999999999996e-05 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350538+0000", 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2625e-05 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:75175 4.11 e80/69 ondisk, result = 0)", 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 68.124295020000005, 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060754698000000003, 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289790+0000", 2024-09-16T13:30:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289791+0000", 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289795+0000", 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289805+0000", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8640000000000008e-06 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.289812+0000", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0419999999999997e-06 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350426+0000", 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.060614384 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350436+0000", 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0412e-05 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.350545+0000", 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108578 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.522 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:75303 4.14 4:2b002a2e:::rbd_data.11a9cafed80f.0000000000000253:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.901261796, 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.063660295000000006, 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 75303 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:32.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512824+0000", 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512828+0000", 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7050000000000001e-06 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512837+0000", 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512842+0000", 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.78e-06 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512881+0000", 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9304000000000002e-05 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.512965+0000", 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4109000000000004e-05 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.518193+0000", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00522814 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576243+0000", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058049798999999999 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576403+0000", 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000159495 2024-09-16T13:30:32.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576442+0000", 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9675999999999998e-05 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:24.576484+0000", 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1517000000000002e-05 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:80319 4.11 4:89d5d412:::rbd_data.11a9ec63c06f.0000000000000002:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.421688717999999, 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065910514000000003, 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 80319 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.529 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992397+0000", 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:30:32.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992401+0000", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7890000000000001e-06 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992410+0000", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.439e-06 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992413+0000", 2024-09-16T13:30:32.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992459+0000", 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5938e-05 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.992538+0000", 2024-09-16T13:30:32.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9200000000000001e-05 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:27.996208+0000", 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036694100000000001 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058221+0000", 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062013453000000003 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:32.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058229+0000", 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7579999999999992e-06 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058266+0000", 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7101000000000003e-05 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:28.058307+0000", 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1171000000000003e-05 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.534 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.4521.0:103535 4.1d 4:beb515f0:::rbd_data.11a976e61a66.000000000000003b:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.543972341, 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099097847000000003, 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2024-09-16T13:30:32.535 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.4521", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 103535 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870113+0000", 2024-09-16T13:30:32.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870115+0000", 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5570000000000001e-06 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870122+0000", 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2620000000000001e-06 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870137+0000", 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5671e-05 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870143+0000", 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870186+0000", 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2833000000000001e-05 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.870282+0000", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5989000000000005e-05 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.875543+0000", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052610900000000004 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969052+0000", 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.093508617000000002 2024-09-16T13:30:32.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969103+0000", 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1416999999999999e-05 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969165+0000", 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1927999999999994e-05 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969211+0000", 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5970999999999997e-05 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(client.4521.0:103535 4.1d e99/32 ondisk, result = 0)", 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.526838040999998, 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081971593999999995, 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2024-09-16T13:30:32.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887247+0000", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887248+0000", 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3200000000000004e-07 2024-09-16T13:30:32.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887252+0000", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5099999999999999e-06 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887263+0000", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1398e-05 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.887269+0000", 2024-09-16T13:30:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969085+0000", 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081815984999999994 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969095+0000", 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8900000000000002e-06 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2024-09-16T13:30:32.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2024-09-16T13:29:45.969219+0000", 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000124213 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T13:30:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:30:32.545 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:32.415+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:32.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:32.547 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:32.546+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:32.622 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:32.623 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:32.634 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-16T13:30:31.925903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209598428,"num_objects":529,"num_object_clones":88,"num_object_copies":1058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":373,"num_read":164984,"num_read_kb":803272,"num_write":21578,"num_write_kb":462493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":605,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":7,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18129,"ondisk_log_size":18129,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2440852,"kb_used_data":28696,"kb_used_omap":13,"kb_used_meta":2412018,"kb_avail":752533868,"statfs":{"total":773094113280,"available":770594680832,"internally_reserved":0,"allocated":29384704,"data_stored":196325057,"data_compressed":3324192,"data_compressed_allocated":14458880,"data_compressed_original":188757732,"omap_allocated":13632,"internal_metadata":2469907136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-175357986,"num_objects":-265,"num_object_clones":-70,"num_object_copies":-530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-265,"num_whiteouts":-61,"num_read":1708,"num_read_kb":1460,"num_write":325,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.864752"},"pg_stats":[{"pgid":"4.1f","version":"138'359","reported_seq":2204,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.066671+0000","last_change":"2024-09-16T13:30:28.066995+0000","last_active":"2024-09-16T13:30:28.066671+0000","last_peered":"2024-09-16T13:30:28.066671+0000","last_clean":"2024-09-16T13:30:28.066671+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:28.066671+0000","last_undegraded":"2024-09-16T13:30:28.066671+0000","last_fullsized":"2024-09-16T13:30:28.066671+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087917699999999998,"stat_sum":{"num_bytes":4505270,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":12,"num_read":2818,"num_read_kb":20109,"num_write":351,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.1e","version":"139'1092","reported_seq":11792,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.100930+0000","last_change":"2024-09-16T13:30:28.101233+0000","last_active":"2024-09-16T13:30:28.100930+0000","last_peered":"2024-09-16T13:30:28.100930+0000","last_clean":"2024-09-16T13:30:28.100930+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:28.100930+0000","last_undegraded":"2024-09-16T13:30:28.100930+0000","last_fullsized":"2024-09-16T13:30:28.100930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1092,"log_dups_size":0,"ondisk_log_size":1092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086399730000000004,"stat_sum":{"num_bytes":524337,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":11,"num_read":15003,"num_read_kb":31076,"num_write":1667,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.1d","version":"141'441","reported_seq":1632,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.155134+0000","last_change":"2024-09-16T13:30:29.074282+0000","last_active":"2024-09-16T13:30:29.155134+0000","last_peered":"2024-09-16T13:30:29.155134+0000","last_clean":"2024-09-16T13:30:29.155134+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:29.155134+0000","last_undegraded":"2024-09-16T13:30:29.155134+0000","last_fullsized":"2024-09-16T13:30:29.155134+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001322847,"stat_sum":{"num_bytes":2386144,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":13,"num_read":2055,"num_read_kb":8165,"num_write":448,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.1c","version":"140'494","reported_seq":4515,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.114723+0000","last_change":"2024-09-16T13:30:28.064485+0000","last_active":"2024-09-16T13:30:28.114723+0000","last_peered":"2024-09-16T13:30:28.114723+0000","last_clean":"2024-09-16T13:30:28.114723+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:28.114723+0000","last_undegraded":"2024-09-16T13:30:28.114723+0000","last_fullsized":"2024-09-16T13:30:28.114723+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044282600000000001,"stat_sum":{"num_bytes":4560432,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":7,"num_read":7611,"num_read_kb":26911,"num_write":503,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.1b","version":"141'477","reported_seq":2337,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.072150+0000","last_change":"2024-09-16T13:30:29.072150+0000","last_active":"2024-09-16T13:30:29.072150+0000","last_peered":"2024-09-16T13:30:29.072150+0000","last_clean":"2024-09-16T13:30:29.072150+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:29.072150+0000","last_undegraded":"2024-09-16T13:30:29.072150+0000","last_fullsized":"2024-09-16T13:30:29.072150+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0048240569999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.1a","version":"141'416","reported_seq":4263,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.166059+0000","last_change":"2024-09-16T13:30:29.071617+0000","last_active":"2024-09-16T13:30:29.166059+0000","last_peered":"2024-09-16T13:30:29.166059+0000","last_clean":"2024-09-16T13:30:29.166059+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:29.166059+0000","last_undegraded":"2024-09-16T13:30:29.166059+0000","last_fullsized":"2024-09-16T13:30:29.166059+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042769770000000004,"stat_sum":{"num_bytes":10241930,"num_objects":16,"num_object_clones":1,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":10,"num_read":4102,"num_read_kb":16545,"num_write":456,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.19","version":"140'398","reported_seq":3465,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.072760+0000","last_change":"2024-09-16T13:30:29.073076+0000","last_active":"2024-09-16T13:30:29.072760+0000","last_peered":"2024-09-16T13:30:29.072760+0000","last_clean":"2024-09-16T13:30:29.072760+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:29.072760+0000","last_undegraded":"2024-09-16T13:30:29.072760+0000","last_fullsized":"2024-09-16T13:30:29.072760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045761100000000002,"stat_sum":{"num_bytes":4799986,"num_objects":18,"num_object_clones":3,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":12,"num_read":3419,"num_read_kb":15850,"num_write":394,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.18","version":"141'381","reported_seq":3432,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.071026+0000","last_change":"2024-09-16T13:30:29.071026+0000","last_active":"2024-09-16T13:30:29.071026+0000","last_peered":"2024-09-16T13:30:29.071026+0000","last_clean":"2024-09-16T13:30:29.071026+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:29.071026+0000","last_undegraded":"2024-09-16T13:30:29.071026+0000","last_fullsized":"2024-09-16T13:30:29.071026+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0037360510000000002,"stat_sum":{"num_bytes":19979422,"num_objects":18,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":9,"num_read":3542,"num_read_kb":32887,"num_write":391,"num_write_kb":30744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.17","version":"141'406","reported_seq":3220,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.077410+0000","last_change":"2024-09-16T13:30:29.077643+0000","last_active":"2024-09-16T13:30:29.077410+0000","last_peered":"2024-09-16T13:30:29.077410+0000","last_clean":"2024-09-16T13:30:29.077410+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:29.077410+0000","last_undegraded":"2024-09-16T13:30:29.077410+0000","last_fullsized":"2024-09-16T13:30:29.077410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.004469041,"stat_sum":{"num_bytes":6815744,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":18,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.16","version":"138'651","reported_seq":2833,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.101219+0000","last_change":"2024-09-16T13:30:28.101443+0000","last_active":"2024-09-16T13:30:28.101219+0000","last_peered":"2024-09-16T13:30:28.101219+0000","last_clean":"2024-09-16T13:30:28.101219+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:28.101219+0000","last_undegraded":"2024-09-16T13:30:28.101219+0000","last_fullsized":"2024-09-16T13:30:28.101219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087526940000000001,"stat_sum":{"num_bytes":688128,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":8,"num_read":4642,"num_read_kb":15931,"num_write":790,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.15","version":"140'826","reported_seq":4108,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.078011+0000","last_change":"2024-09-16T13:30:29.078219+0000","last_active":"2024-09-16T13:30:29.078011+0000","last_peered":"2024-09-16T13:30:29.078011+0000","last_clean":"2024-09-16T13:30:29.078011+0000","last_became_active":"2024-09-16T13:29:55.466429+0000","last_became_peered":"2024-09-16T13:29:55.466429+0000","last_unstale":"2024-09-16T13:30:29.078011+0000","last_undegraded":"2024-09-16T13:30:29.078011+0000","last_fullsized":"2024-09-16T13:30:29.078011+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:31.786154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030206299999999998,"stat_sum":{"num_bytes":32839363,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":13,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.14","version":"140'480","reported_seq":3463,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.074002+0000","last_change":"2024-09-16T13:30:29.074265+0000","last_active":"2024-09-16T13:30:29.074002+0000","last_peered":"2024-09-16T13:30:29.074002+0000","last_clean":"2024-09-16T13:30:29.074002+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:29.074002+0000","last_undegraded":"2024-09-16T13:30:29.074002+0000","last_fullsized":"2024-09-16T13:30:29.074002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001426975,"stat_sum":{"num_bytes":1540218,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":14,"num_read":4924,"num_read_kb":20481,"num_write":503,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.13","version":"141'370","reported_seq":1976,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.079569+0000","last_change":"2024-09-16T13:30:29.079772+0000","last_active":"2024-09-16T13:30:29.079569+0000","last_peered":"2024-09-16T13:30:29.079569+0000","last_clean":"2024-09-16T13:30:29.079569+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:29.079569+0000","last_undegraded":"2024-09-16T13:30:29.079569+0000","last_fullsized":"2024-09-16T13:30:29.079569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.012279887999999999,"stat_sum":{"num_bytes":9458245,"num_objects":17,"num_object_clones":2,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":12,"num_read":1521,"num_read_kb":22420,"num_write":347,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.12","version":"138'805","reported_seq":4202,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.064318+0000","last_change":"2024-09-16T13:30:28.064695+0000","last_active":"2024-09-16T13:30:28.064318+0000","last_peered":"2024-09-16T13:30:28.064318+0000","last_clean":"2024-09-16T13:30:28.064318+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:28.064318+0000","last_undegraded":"2024-09-16T13:30:28.064318+0000","last_fullsized":"2024-09-16T13:30:28.064318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049702999999999998,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":8690,"num_read_kb":20618,"num_write":974,"num_write_kb":8823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.11","version":"141'757","reported_seq":4791,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.176168+0000","last_change":"2024-09-16T13:30:29.073739+0000","last_active":"2024-09-16T13:30:29.176168+0000","last_peered":"2024-09-16T13:30:29.176168+0000","last_clean":"2024-09-16T13:30:29.176168+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:29.176168+0000","last_undegraded":"2024-09-16T13:30:29.176168+0000","last_fullsized":"2024-09-16T13:30:29.176168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001309403,"stat_sum":{"num_bytes":2298191,"num_objects":18,"num_object_clones":3,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":12,"num_read":5906,"num_read_kb":16217,"num_write":945,"num_write_kb":25099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.10","version":"138'758","reported_seq":3893,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.072709+0000","last_change":"2024-09-16T13:30:29.073005+0000","last_active":"2024-09-16T13:30:29.072709+0000","last_peered":"2024-09-16T13:30:29.072709+0000","last_clean":"2024-09-16T13:30:29.072709+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:29.072709+0000","last_undegraded":"2024-09-16T13:30:29.072709+0000","last_fullsized":"2024-09-16T13:30:29.072709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058115499999999997,"stat_sum":{"num_bytes":557056,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":12,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.060367+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:27.060367+0000","last_peered":"2024-09-16T13:30:27.060367+0000","last_clean":"2024-09-16T13:30:27.060367+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:27.060367+0000","last_undegraded":"2024-09-16T13:30:27.060367+0000","last_fullsized":"2024-09-16T13:30:27.060367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"138'490","reported_seq":3572,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.067716+0000","last_change":"2024-09-16T13:30:29.067908+0000","last_active":"2024-09-16T13:30:29.067716+0000","last_peered":"2024-09-16T13:30:29.067716+0000","last_clean":"2024-09-16T13:30:29.067716+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:29.067716+0000","last_undegraded":"2024-09-16T13:30:29.067716+0000","last_fullsized":"2024-09-16T13:30:29.067716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060583499999999999,"stat_sum":{"num_bytes":10227947,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":13,"num_read":3521,"num_read_kb":37836,"num_write":591,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.059127+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:27.059127+0000","last_peered":"2024-09-16T13:30:27.059127+0000","last_clean":"2024-09-16T13:30:27.059127+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:27.059127+0000","last_undegraded":"2024-09-16T13:30:27.059127+0000","last_fullsized":"2024-09-16T13:30:27.059127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"139'342","reported_seq":2589,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.073365+0000","last_change":"2024-09-16T13:30:29.073823+0000","last_active":"2024-09-16T13:30:29.073365+0000","last_peered":"2024-09-16T13:30:29.073365+0000","last_clean":"2024-09-16T13:30:29.073365+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:29.073365+0000","last_undegraded":"2024-09-16T13:30:29.073365+0000","last_fullsized":"2024-09-16T13:30:29.073365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001235613,"stat_sum":{"num_bytes":1104243,"num_objects":18,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":13,"num_read":2287,"num_read_kb":15219,"num_write":414,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"2.5","version":"0'0","reported_seq":115,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.060207+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:27.060207+0000","last_peered":"2024-09-16T13:30:27.060207+0000","last_clean":"2024-09-16T13:30:27.060207+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:27.060207+0000","last_undegraded":"2024-09-16T13:30:27.060207+0000","last_fullsized":"2024-09-16T13:30:27.060207+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"140'789","reported_seq":4147,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.139932+0000","last_change":"2024-09-16T13:30:28.100868+0000","last_active":"2024-09-16T13:30:28.139932+0000","last_peered":"2024-09-16T13:30:28.139932+0000","last_clean":"2024-09-16T13:30:28.139932+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:28.139932+0000","last_undegraded":"2024-09-16T13:30:28.139932+0000","last_fullsized":"2024-09-16T13:30:28.139932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0086265290000000008,"stat_sum":{"num_bytes":18731358,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":15,"num_read":4997,"num_read_kb":49592,"num_write":878,"num_write_kb":31561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.058750+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:27.058750+0000","last_peered":"2024-09-16T13:30:27.058750+0000","last_clean":"2024-09-16T13:30:27.058750+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:27.058750+0000","last_undegraded":"2024-09-16T13:30:27.058750+0000","last_fullsized":"2024-09-16T13:30:27.058750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"138'588","reported_seq":3442,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.064695+0000","last_change":"2024-09-16T13:30:28.064923+0000","last_active":"2024-09-16T13:30:28.064695+0000","last_peered":"2024-09-16T13:30:28.064695+0000","last_clean":"2024-09-16T13:30:28.064695+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:28.064695+0000","last_undegraded":"2024-09-16T13:30:28.064695+0000","last_fullsized":"2024-09-16T13:30:28.064695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059587300000000002,"stat_sum":{"num_bytes":2123740,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":4,"num_read":6327,"num_read_kb":18344,"num_write":656,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"3.3","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.074411+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:28.074411+0000","last_peered":"2024-09-16T13:30:28.074411+0000","last_clean":"2024-09-16T13:30:28.074411+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:28.074411+0000","last_undegraded":"2024-09-16T13:30:28.074411+0000","last_fullsized":"2024-09-16T13:30:28.074411+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":58,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.059300+0000","last_change":"2024-09-16T13:29:55.469146+0000","last_active":"2024-09-16T13:30:27.059300+0000","last_peered":"2024-09-16T13:30:27.059300+0000","last_clean":"2024-09-16T13:30:27.059300+0000","last_became_active":"2024-09-16T13:29:55.468831+0000","last_became_peered":"2024-09-16T13:29:55.468831+0000","last_unstale":"2024-09-16T13:30:27.059300+0000","last_undegraded":"2024-09-16T13:30:27.059300+0000","last_fullsized":"2024-09-16T13:30:27.059300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:56.220967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"139'364","reported_seq":4513,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.064743+0000","last_change":"2024-09-16T13:30:28.064998+0000","last_active":"2024-09-16T13:30:28.064743+0000","last_peered":"2024-09-16T13:30:28.064743+0000","last_clean":"2024-09-16T13:30:28.064743+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:28.064743+0000","last_undegraded":"2024-09-16T13:30:28.064743+0000","last_fullsized":"2024-09-16T13:30:28.064743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048376499999999999,"stat_sum":{"num_bytes":4161536,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":5,"num_read":4249,"num_read_kb":35947,"num_write":392,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"3.0","version":"0'0","reported_seq":121,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.058539+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:27.058539+0000","last_peered":"2024-09-16T13:30:27.058539+0000","last_clean":"2024-09-16T13:30:27.058539+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:27.058539+0000","last_undegraded":"2024-09-16T13:30:27.058539+0000","last_fullsized":"2024-09-16T13:30:27.058539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":132,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.061877+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:27.061877+0000","last_peered":"2024-09-16T13:30:27.061877+0000","last_clean":"2024-09-16T13:30:27.061877+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:27.061877+0000","last_undegraded":"2024-09-16T13:30:27.061877+0000","last_fullsized":"2024-09-16T13:30:27.061877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"138'479","reported_seq":3417,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.064274+0000","last_change":"2024-09-16T13:30:28.064623+0000","last_active":"2024-09-16T13:30:28.064274+0000","last_peered":"2024-09-16T13:30:28.064274+0000","last_clean":"2024-09-16T13:30:28.064274+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:28.064274+0000","last_undegraded":"2024-09-16T13:30:28.064274+0000","last_fullsized":"2024-09-16T13:30:28.064274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045370499999999999,"stat_sum":{"num_bytes":2752645,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":10,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"3.1","version":"105'2","reported_seq":125,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.058425+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:27.058425+0000","last_peered":"2024-09-16T13:30:27.058425+0000","last_clean":"2024-09-16T13:30:27.058425+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:27.058425+0000","last_undegraded":"2024-09-16T13:30:27.058425+0000","last_fullsized":"2024-09-16T13:30:27.058425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":83,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":82,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.063290+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:28.063290+0000","last_peered":"2024-09-16T13:30:28.063290+0000","last_clean":"2024-09-16T13:30:28.063290+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:28.063290+0000","last_undegraded":"2024-09-16T13:30:28.063290+0000","last_fullsized":"2024-09-16T13:30:28.063290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"140'563","reported_seq":3919,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.105724+0000","last_change":"2024-09-16T13:30:28.106035+0000","last_active":"2024-09-16T13:30:28.105724+0000","last_peered":"2024-09-16T13:30:28.105724+0000","last_clean":"2024-09-16T13:30:28.105724+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:28.105724+0000","last_undegraded":"2024-09-16T13:30:28.105724+0000","last_fullsized":"2024-09-16T13:30:28.105724+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.013173638,"stat_sum":{"num_bytes":6441166,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":5362,"num_read_kb":40892,"num_write":530,"num_write_kb":30179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"3.2","version":"101'3","reported_seq":127,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.092215+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:28.092215+0000","last_peered":"2024-09-16T13:30:28.092215+0000","last_clean":"2024-09-16T13:30:28.092215+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:28.092215+0000","last_undegraded":"2024-09-16T13:30:28.092215+0000","last_fullsized":"2024-09-16T13:30:28.092215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":85,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":49,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T13:30:27.061749+0000","last_change":"2024-09-16T13:29:55.466745+0000","last_active":"2024-09-16T13:30:27.061749+0000","last_peered":"2024-09-16T13:30:27.061749+0000","last_clean":"2024-09-16T13:30:27.061749+0000","last_became_active":"2024-09-16T13:29:55.466418+0000","last_became_peered":"2024-09-16T13:29:55.466418+0000","last_unstale":"2024-09-16T13:30:27.061749+0000","last_undegraded":"2024-09-16T13:30:27.061749+0000","last_fullsized":"2024-09-16T13:30:27.061749+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:33.757459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":88,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.063212+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:28.063212+0000","last_peered":"2024-09-16T13:30:28.063212+0000","last_clean":"2024-09-16T13:30:28.063212+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:28.063212+0000","last_undegraded":"2024-09-16T13:30:28.063212+0000","last_fullsized":"2024-09-16T13:30:28.063212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"139'593","reported_seq":2462,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.066098+0000","last_change":"2024-09-16T13:30:28.066613+0000","last_active":"2024-09-16T13:30:28.066098+0000","last_peered":"2024-09-16T13:30:28.066098+0000","last_clean":"2024-09-16T13:30:28.066098+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:28.066098+0000","last_undegraded":"2024-09-16T13:30:28.066098+0000","last_fullsized":"2024-09-16T13:30:28.066098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073981299999999997,"stat_sum":{"num_bytes":5511750,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":10,"num_read":3484,"num_read_kb":17249,"num_write":726,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.8","version":"141'1248","reported_seq":5704,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.078014+0000","last_change":"2024-09-16T13:30:29.078014+0000","last_active":"2024-09-16T13:30:29.078014+0000","last_peered":"2024-09-16T13:30:29.078014+0000","last_clean":"2024-09-16T13:30:29.078014+0000","last_became_active":"2024-09-16T13:29:55.466349+0000","last_became_peered":"2024-09-16T13:29:55.466349+0000","last_unstale":"2024-09-16T13:30:29.078014+0000","last_undegraded":"2024-09-16T13:30:29.078014+0000","last_fullsized":"2024-09-16T13:30:29.078014+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:08.619344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000363999,"stat_sum":{"num_bytes":4540824,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":6,"num_read":9582,"num_read_kb":25606,"num_write":1821,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.9","version":"140'385","reported_seq":4767,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.067357+0000","last_change":"2024-09-16T13:30:29.067658+0000","last_active":"2024-09-16T13:30:29.067357+0000","last_peered":"2024-09-16T13:30:29.067357+0000","last_clean":"2024-09-16T13:30:29.067357+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:29.067357+0000","last_undegraded":"2024-09-16T13:30:29.067357+0000","last_fullsized":"2024-09-16T13:30:29.067357+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039516900000000001,"stat_sum":{"num_bytes":14825370,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":5854,"num_read_kb":30067,"num_write":439,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.a","version":"140'557","reported_seq":4538,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.161135+0000","last_change":"2024-09-16T13:30:28.072130+0000","last_active":"2024-09-16T13:30:28.161135+0000","last_peered":"2024-09-16T13:30:28.161135+0000","last_clean":"2024-09-16T13:30:28.161135+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:28.161135+0000","last_undegraded":"2024-09-16T13:30:28.161135+0000","last_fullsized":"2024-09-16T13:30:28.161135+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0062577079999999998,"stat_sum":{"num_bytes":2654208,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":14,"num_read":6013,"num_read_kb":43657,"num_write":583,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.b","version":"140'420","reported_seq":5444,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.101220+0000","last_change":"2024-09-16T13:30:28.101455+0000","last_active":"2024-09-16T13:30:28.101220+0000","last_peered":"2024-09-16T13:30:28.101220+0000","last_clean":"2024-09-16T13:30:28.101220+0000","last_became_active":"2024-09-16T13:28:26.985683+0000","last_became_peered":"2024-09-16T13:28:26.985683+0000","last_unstale":"2024-09-16T13:30:28.101220+0000","last_undegraded":"2024-09-16T13:30:28.101220+0000","last_fullsized":"2024-09-16T13:30:28.101220+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:03.369077+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0093649179999999999,"stat_sum":{"num_bytes":8216503,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":11,"num_read":5833,"num_read_kb":40471,"num_write":493,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.c","version":"140'478","reported_seq":5424,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.072200+0000","last_change":"2024-09-16T13:30:29.072425+0000","last_active":"2024-09-16T13:30:29.072200+0000","last_peered":"2024-09-16T13:30:29.072200+0000","last_clean":"2024-09-16T13:30:29.072200+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:29.072200+0000","last_undegraded":"2024-09-16T13:30:29.072200+0000","last_fullsized":"2024-09-16T13:30:29.072200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004889343,"stat_sum":{"num_bytes":40,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":16,"num_read":6489,"num_read_kb":25242,"num_write":545,"num_write_kb":8621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.d","version":"139'445","reported_seq":3120,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.092193+0000","last_change":"2024-09-16T13:30:28.092508+0000","last_active":"2024-09-16T13:30:28.092193+0000","last_peered":"2024-09-16T13:30:28.092193+0000","last_clean":"2024-09-16T13:30:28.092193+0000","last_became_active":"2024-09-16T13:29:55.469723+0000","last_became_peered":"2024-09-16T13:29:55.469723+0000","last_unstale":"2024-09-16T13:30:28.092193+0000","last_undegraded":"2024-09-16T13:30:28.092193+0000","last_fullsized":"2024-09-16T13:30:28.092193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.171374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043792499999999998,"stat_sum":{"num_bytes":9117252,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":16,"num_read":2952,"num_read_kb":22132,"num_write":462,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9","length":"9"}]},{"pgid":"4.e","version":"138'361","reported_seq":1760,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T13:30:29.072481+0000","last_change":"2024-09-16T13:30:29.072655+0000","last_active":"2024-09-16T13:30:29.072481+0000","last_peered":"2024-09-16T13:30:29.072481+0000","last_clean":"2024-09-16T13:30:29.072481+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:29.072481+0000","last_undegraded":"2024-09-16T13:30:29.072481+0000","last_fullsized":"2024-09-16T13:30:29.072481+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050980230000000001,"stat_sum":{"num_bytes":10024775,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":19,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8","length":"a"}]},{"pgid":"4.f","version":"139'836","reported_seq":3386,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T13:30:28.063968+0000","last_change":"2024-09-16T13:30:28.064228+0000","last_active":"2024-09-16T13:30:28.063968+0000","last_peered":"2024-09-16T13:30:28.063968+0000","last_clean":"2024-09-16T13:30:28.063968+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:28.063968+0000","last_undegraded":"2024-09-16T13:30:28.063968+0000","last_fullsized":"2024-09-16T13:30:28.063968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036146399999999998,"stat_sum":{"num_bytes":7380050,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":14,"num_read":5502,"num_read_kb":14714,"num_write":1121,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"9"}]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":209007873,"num_objects":520,"num_object_clones":88,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":370,"num_read":164918,"num_read_kb":803217,"num_write":21473,"num_write_kb":460861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11427840,"data_stored":119450894,"data_compressed":2492082,"data_compressed_allocated":10051584,"data_compressed_original":118752114,"omap_allocated":0,"internal_metadata":0},"log_size":18049,"ondisk_log_size":18049,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":168,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509473,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304720,"kb_used_data":3204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067120,"statfs":{"total":96636764160,"available":96324730880,"internally_reserved":0,"allocated":3280896,"data_stored":18962272,"data_compressed":268502,"data_compressed_allocated":1470464,"data_compressed_original":18042880,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.307},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.32800000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39000000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.40600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305424,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066416,"statfs":{"total":96636764160,"available":96324009984,"internally_reserved":0,"allocated":3997696,"data_stored":27003425,"data_compressed":454385,"data_compressed_allocated":2064384,"data_compressed_original":26013696,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.441},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.35299999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.31},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.504},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.68799999999999994}]}]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301196,"kb_used_data":3832,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070644,"statfs":{"total":96636764160,"available":96328339456,"internally_reserved":0,"allocated":3923968,"data_stored":24813328,"data_compressed":582777,"data_compressed_allocated":1978368,"data_compressed_original":23831410,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33000000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308804,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063036,"statfs":{"total":96636764160,"available":96320548864,"internally_reserved":0,"allocated":3203072,"data_stored":16911815,"data_compressed":418570,"data_compressed_allocated":1376256,"data_compressed_original":16003072,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.313}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.376}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.29099999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.433},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.439},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.39300000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542179,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301320,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070520,"statfs":{"total":96636764160,"available":96328212480,"internally_reserved":0,"allocated":4050944,"data_stored":38045316,"data_compressed":464762,"data_compressed_allocated":2203648,"data_compressed_original":37102450,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.61099999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.56200000000000006}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.442},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.624},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65300000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542177,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307932,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063908,"statfs":{"total":96636764160,"available":96321441792,"internally_reserved":0,"allocated":2306048,"data_stored":5516874,"data_compressed":251115,"data_compressed_allocated":512000,"data_compressed_original":4599808,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.26500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.33700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.46000000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.46999999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311048,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94060792,"statfs":{"total":96636764160,"available":96318251008,"internally_reserved":0,"allocated":5496832,"data_stored":46473942,"data_compressed":653269,"data_compressed_allocated":3543040,"data_compressed_original":45498368,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.25},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.318},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.31}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.37}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.503}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.47399999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.433}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300408,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071432,"statfs":{"total":96636764160,"available":96329146368,"internally_reserved":0,"allocated":3125248,"data_stored":18598085,"data_compressed":230812,"data_compressed_allocated":1310720,"data_compressed_original":17666048,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.312},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.505},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.627}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.62},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":83,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":85,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1392640,"data_stored":17158798,"data_compressed":225213,"data_compressed_allocated":1273856,"data_compressed_original":17092608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1708032,"data_stored":16794920,"data_compressed":390298,"data_compressed_allocated":1482752,"data_compressed_original":16687104,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":4587748,"data_compressed":202085,"data_compressed_allocated":462848,"data_compressed_original":4526080,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1134592,"data_stored":10777585,"data_compressed":296657,"data_compressed_allocated":1003520,"data_compressed_original":10727424,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1851392,"data_stored":17866669,"data_compressed":503619,"data_compressed_allocated":1605632,"data_compressed_original":17747968,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2150400,"data_stored":23187956,"data_compressed":449736,"data_compressed_allocated":1912832,"data_compressed_original":23052288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":12764658,"data_compressed":183303,"data_compressed_allocated":978944,"data_compressed_original":12673906,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1458176,"data_stored":16312560,"data_compressed":241171,"data_compressed_allocated":1331200,"data_compressed_original":16244736,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:32.637 INFO:tasks.thrashosds.thrasher:Setting 4.b to [6, 4] 2024-09-16T13:30:32.637 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.b', '6', '4'] 2024-09-16T13:30:32.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.b 6 4 2024-09-16T13:30:32.647 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:32.647 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:32.647 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:32.647 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:32.648 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:32.646+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:32.655 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:32.749 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:32.747+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:32.795 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:32.795 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:32.795 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:32.795 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:32.795 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:32.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:32.887 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:32.885+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:32.931 INFO:teuthology.orchestra.run.smithi076.stderr:set 4.b pg_upmap mapping to [6,4] 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:7613 4.d 4:b52d880f:::rbd_data.11a91a08d9ca.0000000000000097:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 115.232762592, 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071705493999999995, 2024-09-16T13:30:32.946 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 7613 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.947 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710339+0000", 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710340+0000", 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2099999999999995e-07 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710344+0000", 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4889999999999997e-06 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.948 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710356+0000", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2121e-05 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710361+0000", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3680000000000004e-06 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710419+0000", 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8341999999999998e-05 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.949 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.710503+0000", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4352999999999997e-05 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.713969+0000", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003465096 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781919+0000", 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067950694000000006 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.950 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781932+0000", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3071000000000001e-05 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781990+0000", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7958000000000003e-05 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.782044+0000", 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4080999999999999e-05 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.951 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:7613 4.d e41/32 ondisk, result = 0)", 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 115.22322348100001, 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062044426, 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.952 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719878+0000", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719880+0000", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719885+0000", 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.609e-06 2024-09-16T13:30:32.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719902+0000", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7133999999999999e-05 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.719911+0000", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1649999999999995e-06 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781901+0000", 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.061989814999999997 2024-09-16T13:30:32.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781911+0000", 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8069999999999994e-06 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.781922+0000", 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1246e-05 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25675 4.1a 4:5fe3708c:::rbd_data.11a926bfcc53.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:32.955 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 106.256766746, 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043306797000000001, 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25675 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:32.956 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686335+0000", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0400000000000005e-07 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686340+0000", 2024-09-16T13:30:32.957 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4599999999999996e-06 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686354+0000", 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3709e-05 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686358+0000", 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3900000000000003e-06 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.958 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686396+0000", 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7555000000000002e-05 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686440+0000", 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4212999999999998e-05 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.688580+0000", 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021405230000000001 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.959 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729543+0000", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040962795000000003 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729575+0000", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1884999999999997e-05 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729615+0000", 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.960 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.729641+0000", 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6367999999999999e-05 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:28352 4.13 4:c90e7765:::rbd_data.11a9320547a2.000000000000000c:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 101.546355295, 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055810838000000002, 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.961 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28352 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.962 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396746+0000", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396747+0000", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4799999999999997e-07 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396752+0000", 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7190000000000001e-06 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.963 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396762+0000", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0081000000000001e-05 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396766+0000", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0520000000000003e-06 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396805+0000", 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8872000000000003e-05 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.964 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.396853+0000", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8649000000000001e-05 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.399730+0000", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028770879999999999 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452419+0000", 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052688832999999997 2024-09-16T13:30:32.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452493+0000", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3486999999999996e-05 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452532+0000", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8887000000000002e-05 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452557+0000", 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5422000000000002e-05 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.966 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:28352 4.13 e51/32 ondisk, result = 0)", 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 101.535708246, 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045169504999999999, 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.967 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407393+0000", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407394+0000", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407398+0000", 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.968 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407409+0000", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0698e-05 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.407413+0000", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.481e-06 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452482+0000", 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045068911000000003 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.969 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452488+0000", 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4110000000000001e-06 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:51.452563+0000", 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4326000000000006e-05 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:38134 4.1b 4:df6a86c3:::rbd_data.11a96b196269.0000000000000001:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T13:30:32.970 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 92.868646665, 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046189656000000003, 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 38134 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:32.971 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074455+0000", 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6100000000000001e-07 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074462+0000", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7870000000000004e-06 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074469+0000", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7009999999999999e-06 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074472+0000", 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.163e-06 2024-09-16T13:30:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074508+0000", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5759000000000001e-05 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.074552+0000", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4540999999999999e-05 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.077327+0000", 2024-09-16T13:30:32.974 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027748629999999998 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120575+0000", 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043248083999999999 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120584+0000", 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.354e-06 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.975 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120616+0000", 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2228999999999999e-05 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:00.120644+0000", 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8614000000000001e-05 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:75407 4.c 4:31b8562e:::rbd_data.11a9cafed80f.0000000000000189:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 68.301015609000004, 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.102420451, 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75407 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642086+0000", 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642087+0000", 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.978 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642091+0000", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4560000000000002e-06 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642101+0000", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0005e-05 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642105+0000", 2024-09-16T13:30:32.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4460000000000003e-06 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642137+0000", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1503e-05 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.642220+0000", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2745999999999997e-05 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.646470+0000", 2024-09-16T13:30:32.980 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004249928 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744416+0000", 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.097946737000000006 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744427+0000", 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0456e-05 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.981 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744466+0000", 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9143000000000001e-05 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744506+0000", 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0250999999999998e-05 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.982 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:75407 4.c e80/32 ondisk, result = 0)", 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 68.283882715000004, 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085200724000000005, 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.983 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659219+0000", 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659220+0000", 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.319e-06 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.984 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659226+0000", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.592e-06 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659242+0000", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6342e-05 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.659249+0000", 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1350000000000003e-06 2024-09-16T13:30:32.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744378+0000", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085129120000000003 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744387+0000", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7080000000000003e-06 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:24.744419+0000", 2024-09-16T13:30:32.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2508000000000001e-05 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:80112 4.1a 4:5d4189ab:::rbd_data.11a9ec63c06f.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 65.323493768999995, 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095535219000000005, 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.987 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 80112 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619608+0000", 2024-09-16T13:30:32.988 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619610+0000", 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4899999999999999e-06 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619627+0000", 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7283e-05 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.989 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619657+0000", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9169000000000001e-05 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619668+0000", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1637000000000001e-05 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.619835+0000", 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016670000000000001 2024-09-16T13:30:32.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.620030+0000", 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019503599999999999 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3", 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.624331+0000", 2024-09-16T13:30:32.991 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004300768 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715062+0000", 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.090731265000000005 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715076+0000", 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3589e-05 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.992 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715114+0000", 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8729000000000001e-05 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715143+0000", 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8552999999999999e-05 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.993 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:80112 4.1a e82/32 ondisk, result = 0)", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 65.307005047999994, 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078969757000000002, 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.994 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636096+0000", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636098+0000", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4610000000000001e-06 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636102+0000", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3429999999999998e-06 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:32.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636119+0000", 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6917000000000001e-05 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.636127+0000", 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3210000000000005e-06 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714864+0000", 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.078736303999999993 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.996 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.714881+0000", 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7067000000000001e-05 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:27.715066+0000", 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018534399999999999 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.997 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91793 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 1982464~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.927120011, 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039127990000000001, 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91793 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.998 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.015981+0000", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016483+0000", 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050150499999999996 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:32.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016487+0000", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0600000000000001e-06 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016497+0000", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7289999999999995e-06 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016502+0000", 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2950000000000001e-06 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.054957+0000", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038455169999999997 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055024+0000", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6513000000000005e-05 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055109+0000", 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5717999999999995e-05 2024-09-16T13:30:33.001 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91798 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3211264~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.926527082, 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039257531999999998, 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.002 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91798 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016574+0000", 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016575+0000", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016577+0000", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.905e-06 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016582+0000", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.485e-06 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016585+0000", 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2509999999999999e-06 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055710+0000", 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039124164000000003 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055747+0000", 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7826e-05 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055832+0000", 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4509e-05 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:91799 4.9 4:90f392e6:::rbd_data.11a9f9a8a0c.0000000000000003:27 [read 3457024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.926512418999998, 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039424519999999998, 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.006 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 91799 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.016589+0000", 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017876+0000", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001287137 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017886+0000", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5950000000000005e-06 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017898+0000", 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2289999999999999e-05 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.017904+0000", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0379999999999996e-06 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055865+0000", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037960689999999998 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.055936+0000", 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1280999999999997e-05 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.056014+0000", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7489000000000003e-05 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:97067 4.18 4:1b3fedb0:::rbd_data.11a98c604b63.0000000000000062:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.631935767000002, 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052530698000000001, 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.010 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 97067 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311166+0000", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311170+0000", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.642e-06 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311177+0000", 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7209999999999997e-06 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.012 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311179+0000", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8490000000000002e-06 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311218+0000", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8760000000000002e-05 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.311301+0000", 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3335000000000001e-05 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.316782+0000", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054801049999999999 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363503+0000", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046720985999999999 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363613+0000", 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110666 2024-09-16T13:30:33.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363652+0000", 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9118000000000001e-05 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363696+0000", 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4001000000000001e-05 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:97067 4.18 e91/32 ondisk, result = 0)", 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.618654540999998, 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039255279999999997, 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324447+0000", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324448+0000", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324452+0000", 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:30:33.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324463+0000", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.153e-05 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.324469+0000", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363542+0000", 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039073402 2024-09-16T13:30:33.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363550+0000", 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5299999999999999e-06 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:37.363702+0000", 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015233400000000001 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103858 4.13 4:cb1ec18b:::rbd_data.11a976e61a66.00000000000000e9:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.653404866000002, 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.082006685999999995, 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103858 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289697+0000", 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0800000000000004e-07 2024-09-16T13:30:33.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289702+0000", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2910000000000001e-06 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289712+0000", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0601999999999999e-05 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289717+0000", 2024-09-16T13:30:33.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3619999999999999e-06 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289750+0000", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3022999999999999e-05 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.289835+0000", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5130999999999995e-05 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.294259+0000", 2024-09-16T13:30:33.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044243599999999996 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371513+0000", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.077253646999999995 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371624+0000", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011076700000000001 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371662+0000", 2024-09-16T13:30:33.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8794999999999997e-05 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371703+0000", 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0899999999999998e-05 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103858 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.637128136999998, 2024-09-16T13:30:33.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065542639, 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305973+0000", 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305974+0000", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1399999999999996e-07 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305978+0000", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2010000000000001e-06 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305990+0000", 2024-09-16T13:30:33.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1928e-05 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.305997+0000", 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4989999999999999e-06 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371440+0000", 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.065443108 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371448+0000", 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1259999999999998e-06 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.371516+0000", 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7963e-05 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.029 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103925 4.18 4:1febfdc4:::rbd_data.11a976e61a66.000000000000003e:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.365783223000001, 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052147767999999997, 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103925 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577318+0000", 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577320+0000", 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.497e-06 2024-09-16T13:30:33.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577327+0000", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1300000000000003e-06 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577340+0000", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3576e-05 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577346+0000", 2024-09-16T13:30:33.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5230000000000003e-06 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577396+0000", 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9759000000000003e-05 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577528+0000", 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013241300000000001 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.582053+0000", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045247680000000002 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629344+0000", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047291622999999998 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629360+0000", 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629421+0000", 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0702999999999998e-05 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.629466+0000", 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5294999999999999e-05 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.035 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:103926 4.13 4:ce3c3233:::rbd_data.11a976e61a66.000000000000003f:head [set-alloc-hint object_size 524288 write_size 524288,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.365588328000001, 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071657002999999997, 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 103926 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577513+0000", 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577514+0000", 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577517+0000", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3909999999999998e-06 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577522+0000", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4040000000000003e-06 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577526+0000", 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1099999999999999e-06 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577570+0000", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4950000000000002e-05 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.577699+0000", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000128582 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 6", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.585153+0000", 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074536150000000002 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648936+0000", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.063783461999999999 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649105+0000", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016883499999999999 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649145+0000", 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0207999999999999e-05 2024-09-16T13:30:33.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649170+0000", 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:103926 4.13 e99/32 ondisk, result = 0)", 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.346325409999999, 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052400257999999998, 2024-09-16T13:30:33.041 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596776+0000", 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596777+0000", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.234e-06 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596781+0000", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0949999999999998e-06 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596793+0000", 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1635e-05 2024-09-16T13:30:33.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.596799+0000", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8499999999999999e-06 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648960+0000", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052161066999999998 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.648967+0000", 2024-09-16T13:30:33.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0979999999999996e-06 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:46.649176+0000", 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020927900000000001 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.045 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:33.047 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T13:30:33.045+0000 7fd880df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33362) UID: 0 2024-09-16T13:30:33.048 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:33.148 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:33.146+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:33.187 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:33.187 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-09-16T13:30:33.187 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:33.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:33.249 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:33.247+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:33.293 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:33.293 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-09-16T13:30:33.293 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-09-16T13:30:33.293 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-09-16T13:30:33.293 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:33.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:33.350 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:33.348+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:33.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:3537 4.15 4:ab92d926:::rbd_data.11a976a8123d.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 118.668149731, 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051410741000000003, 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.401 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3537 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730016+0000", 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.730018+0000", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2729999999999999e-06 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733588+0000", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035699600000000001 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733606+0000", 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8555000000000001e-05 2024-09-16T13:30:33.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733614+0000", 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733666+0000", 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1844e-05 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.733776+0000", 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110265 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.737253+0000", 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034768099999999999 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781260+0000", 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044007424000000003 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781311+0000", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1343e-05 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781374+0000", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2518e-05 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:34.781427+0000", 2024-09-16T13:30:33.406 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3325000000000002e-05 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:6344 4.3 e41/32 4:c6fa2f15:::rbd_data.11a9df936547.0000000000000000:head v 41'24, mlcod=40'21)", 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 116.22525840900001, 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.092801716000000006, 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172908+0000", 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.172909+0000", 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7199999999999998e-07 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188867+0000", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015958591000000001 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188910+0000", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2540000000000003e-05 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188919+0000", 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3589999999999997e-06 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188962+0000", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2796000000000002e-05 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.188982+0000", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0169999999999998e-05 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265669+0000", 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.076687195999999999 2024-09-16T13:30:33.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:37.265710+0000", 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0292e-05 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:25676 4.15 4:aa69e2b5:::rbd_data.11a926bfcc53.0000000000000003:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 106.711800381, 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057422313000000003, 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 25676 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686366+0000", 2024-09-16T13:30:33.412 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686368+0000", 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7969999999999999e-06 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686373+0000", 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5860000000000004e-06 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.413 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686387+0000", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3876000000000001e-05 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686393+0000", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5690000000000004e-06 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686432+0000", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9610999999999997e-05 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.686485+0000", 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2327999999999998e-05 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.689006+0000", 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002521536 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743694+0000", 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.054688085999999997 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.415 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743703+0000", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743742+0000", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9461e-05 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743788+0000", 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6116999999999999e-05 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.416 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.4521.0:25676 4.15 e48/32 ondisk, result = 0)", 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 106.70171111400001, 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047241076999999999, 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696455+0000", 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696456+0000", 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2600000000000001e-07 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.418 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696460+0000", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5860000000000001e-06 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696473+0000", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3337e-05 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.696478+0000", 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4909999999999996e-06 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.419 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743681+0000", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047202755999999998 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743689+0000", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:28:46.743696+0000", 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8260000000000004e-06 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:52782 4.8 4:1793f2ca:::rbd_data.11a9fc41ce3c.0000000000000066:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,write 335872~902] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 83.422998086999996, 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94629844500000004, 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.421 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 52782 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975168+0000", 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975169+0000", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.24e-06 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975231+0000", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1323999999999998e-05 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975242+0000", 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1511999999999999e-05 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975247+0000", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.702e-06 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975304+0000", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7213000000000003e-05 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:09.975310+0000", 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2709999999999998e-06 2024-09-16T13:30:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917353+0000", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.94204283899999997 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917497+0000", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000143776 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.917700+0000", 2024-09-16T13:30:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020347900000000001 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.919032+0000", 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331942 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921394+0000", 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023620329999999999 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.426 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921443+0000", 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8962000000000003e-05 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:10.921467+0000", 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3152000000000001e-05 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:58637 4.15 4:a91c0c14:::rbd_data.11a99fc6eff7.00000000000001f9:head [set-alloc-hint object_size 262144 write_size 262144,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:30:33.427 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 78.474839114000005, 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035725795999999997, 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 58637 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923327+0000", 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923329+0000", 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5910000000000001e-06 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923335+0000", 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923348+0000", 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3321e-05 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923354+0000", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923401+0000", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6751e-05 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.923532+0000", 2024-09-16T13:30:33.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000130949 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.926219+0000", 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00268702 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958975+0000", 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032756 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.958983+0000", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3890000000000005e-06 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959015+0000", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1442000000000002e-05 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:14.959053+0000", 2024-09-16T13:30:33.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8259000000000001e-05 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92117 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6094848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387659128999999, 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027584033000000001, 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.434 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92117 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010507+0000", 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010508+0000", 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.82e-07 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010510+0000", 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0080000000000001e-06 2024-09-16T13:30:33.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010517+0000", 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.131e-06 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010521+0000", 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038022+0000", 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027501279999999999 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038056+0000", 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4449999999999997e-05 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038091+0000", 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4721999999999997e-05 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92118 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6340608~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387641531, 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027676334, 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92118 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.439 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010525+0000", 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:30:33.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010527+0000", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.782e-06 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010533+0000", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010537+0000", 2024-09-16T13:30:33.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.49e-06 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038123+0000", 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027586784 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038163+0000", 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9638999999999998e-05 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038201+0000", 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8002e-05 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92119 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6586368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387626832000002, 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027757513000000001, 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.443 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92119 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010539+0000", 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010540+0000", 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5400000000000002e-07 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010541+0000", 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1710000000000001e-06 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010544+0000", 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010546+0000", 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1179999999999999e-06 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038226+0000", 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027679780000000001 2024-09-16T13:30:33.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038263+0000", 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6653000000000003e-05 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038297+0000", 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.43e-05 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92120 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387615900999997, 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027837562, 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92120 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010550+0000", 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010551+0000", 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010552+0000", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6840000000000001e-06 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010558+0000", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7219999999999996e-06 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010562+0000", 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6789999999999998e-06 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038320+0000", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027758069999999999 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038355+0000", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4999000000000002e-05 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038388+0000", 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.451 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92121 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7077888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387600282000001, 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027914596, 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92121 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.452 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010566+0000", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.34e-07 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010573+0000", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6059999999999999e-06 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010580+0000", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010583+0000", 2024-09-16T13:30:33.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.895e-06 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038411+0000", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027827676999999999 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038446+0000", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5006999999999999e-05 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038481+0000", 2024-09-16T13:30:33.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4448e-05 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92122 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7323648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387579019, 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027991562000000001, 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.456 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92122 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010587+0000", 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010588+0000", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010589+0000", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6419999999999999e-06 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010595+0000", 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7320000000000003e-06 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010598+0000", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4769999999999999e-06 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038506+0000", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027908018 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038544+0000", 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7455e-05 2024-09-16T13:30:33.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038579+0000", 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4891000000000002e-05 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92123 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7569408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387564009000002, 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.02807072, 2024-09-16T13:30:33.460 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92123 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010602+0000", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010603+0000", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010604+0000", 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:30:33.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010611+0000", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6490000000000002e-06 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010615+0000", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7079999999999999e-06 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038603+0000", 2024-09-16T13:30:33.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027988128000000001 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038639+0000", 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6115000000000002e-05 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038673+0000", 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4072000000000002e-05 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.464 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92124 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 7815168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387547927999996, 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028144928999999999, 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92124 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.465 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010618+0000", 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010619+0000", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010620+0000", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5489999999999999e-06 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010628+0000", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010632+0000", 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038697+0000", 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028064490000000001 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038730+0000", 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3191000000000001e-05 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038763+0000", 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3200000000000001e-05 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92125 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387508060999998, 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028196029000000001, 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.469 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92125 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010658+0000", 2024-09-16T13:30:33.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010659+0000", 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4200000000000001e-07 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010660+0000", 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.821e-06 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010665+0000", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010667+0000", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5600000000000001e-06 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038786+0000", 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028118975000000001 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038820+0000", 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3756e-05 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038854+0000", 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4149000000000001e-05 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:92126 4.15 4:ab736b70:::rbd_data.11a9f9a8a0c.0000000000000000:27 [read 8306688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 59.387496708999997, 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028278207, 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 92126 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010670+0000", 2024-09-16T13:30:33.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1e-07 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010671+0000", 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010676+0000", 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8829999999999997e-06 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.010678+0000", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038881+0000", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028202847999999999 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038915+0000", 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3751999999999998e-05 2024-09-16T13:30:33.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:34.038948+0000", 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2721000000000001e-05 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100149 4.1f e95/32 4:fcd6cc1f:::rbd_data.11a97e9c0852.0000000000000001:head v 95'272, mlcod=95'270)", 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:33.478 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.662945823000001, 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028307645999999999, 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735220+0000", 2024-09-16T13:30:33.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.735222+0000", 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.063e-06 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736760+0000", 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015378670000000001 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0933000000000001e-05 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736791+0000", 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0029e-05 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736863+0000", 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1792999999999995e-05 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736902+0000", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8804000000000003e-05 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763477+0000", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026574878 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.763528+0000", 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1279000000000002e-05 2024-09-16T13:30:33.482 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4521.0:100152 4.15 4:adcf89f4:::rbd_data.11a97e9c0852.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,truncate 6225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.661657497, 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029139728, 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4521", 2024-09-16T13:30:33.483 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.76:0/1181279960", 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 100152 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736509+0000", 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736514+0000", 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0370000000000002e-06 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741416+0000", 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0049025240000000001 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741433+0000", 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6534000000000001e-05 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741439+0000", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1510000000000001e-06 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741484+0000", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4867999999999998e-05 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741589+0000", 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105355 2024-09-16T13:30:33.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 7", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.741807+0000", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000217965 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.764737+0000", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022929984 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765541+0000", 2024-09-16T13:30:33.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080343000000000005 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765602+0000", 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1728000000000003e-05 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.765648+0000", 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6152000000000001e-05 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.491 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.492 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.492 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100151 4.7 e95/32 4:e674e202:::rbd_data.11a97e9c0852.0000000000000002:head v 95'317, mlcod=91'315)", 2024-09-16T13:30:33.492 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.661385562, 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035473583000000003, 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:33.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.736781+0000", 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8200000000000001e-07 2024-09-16T13:30:33.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746522+0000", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0097406100000000002 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746547+0000", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5426999999999999e-05 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746560+0000", 2024-09-16T13:30:33.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2744e-05 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746628+0000", 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8496000000000003e-05 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.746660+0000", 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2054999999999997e-05 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772214+0000", 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025554009999999999 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.772254+0000", 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9959000000000001e-05 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.4521.0:100154 4.3 e95/32 4:c56a07c9:::rbd_data.11a97e9c0852.0000000000000007:head v 95'448, mlcod=95'446)", 2024-09-16T13:30:33.498 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 51.655906940000001, 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027870256999999999, 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742259+0000", 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-09-16T13:30:33.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.742260+0000", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1599999999999999e-06 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747334+0000", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050737250000000003 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747346+0000", 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1527e-05 2024-09-16T13:30:33.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747352+0000", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4710000000000004e-06 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747436+0000", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4192999999999998e-05 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.747460+0000", 2024-09-16T13:30:33.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4122999999999999e-05 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770100+0000", 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022639705 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-09-16T13:29:41.770130+0000", 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9352999999999998e-05 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-09-16T13:30:33.502 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-09-16T13:30:33.503 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2024-09-16T13:30:33.504 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:30:33.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2024-09-16T13:30:33.505 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:30:33.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:33.505 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:33.502+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:33.516 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2024-09-16T13:30:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:30:33.531 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2024-09-16T13:30:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:30:33.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2024-09-16T13:30:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:30:33.562 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2024-09-16T13:30:33.562 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:30:33.578 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2024-09-16T13:30:33.578 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:30:33.592 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2024-09-16T13:30:33.593 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:30:33.604 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:33.602+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:33.645 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:33.645 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:33.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:33.706 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:33.704+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:33.752 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:33.752 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:33.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:33.808 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:33.806+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:33.861 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:33.862 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:33.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:33.907 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:33.906+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:33.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:30:34.008 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.009 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.009 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T13:30:34.007+0000 7fd6a53f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33327) UID: 0 2024-09-16T13:30:34.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:34.109 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:34.108+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:34.156 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.156 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:34.210 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:34.208+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:34.262 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.262 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:30:34.312 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T13:30:34.310+0000 7f218bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33323) UID: 0 2024-09-16T13:30:34.369 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.369 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.380 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:30:34.424 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:34.422+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:34.477 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.478 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:30:34.527 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T13:30:34.525+0000 7f0d50df8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33345) UID: 0 2024-09-16T13:30:34.614 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:34.612+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:34.623 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.623 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:30:34.716 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T13:30:34.714+0000 7fca4ebf8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33322) UID: 0 2024-09-16T13:30:34.770 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.770 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:30:34.817 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:34.816+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:34.876 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:34.877 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:34.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:30:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:30:34.928 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:34.926+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:34.993 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.049 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:30:35.051 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T13:30:35.049+0000 7f068bff8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33320) UID: 0 2024-09-16T13:30:35.150 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.150 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.150 INFO:tasks.ceph.osd.3.smithi076.stderr:2024-09-16T13:30:35.149+0000 7f3f801f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33361) UID: 0 2024-09-16T13:30:35.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:30:35.252 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T13:30:35.250+0000 7f1e9e7f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33347) UID: 0 2024-09-16T13:30:35.273 INFO:teuthology.orchestra.run.smithi076.stdout:All operations completed A-OK! 2024-09-16T13:30:35.285 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:35.285 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-09-16T13:30:35.297 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T13:30:35.305 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T13:30:35.309 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.309 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:30:35.460 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.460 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:30:35.567 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.568 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:30:35.675 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.675 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:30:35.781 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.782 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:30:35.890 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.891 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:35.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:30:35.997 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:35.997 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.006 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:30:36.107 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.107 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.116 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:30:36.213 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.214 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:30:36.320 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.320 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:30:36.429 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.429 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:30:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.533 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:30:36.640 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.641 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:30:36.747 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.747 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:30:36.854 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.854 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:30:36.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:30:36.960 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:36.960 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:36.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:30:37.068 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-09-16T13:30:37.068 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:30:38.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T13:30:38.112 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2024-09-16T13:30:38.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:30:38.162 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a: {} 2024-09-16T13:30:38.162 INFO:teuthology.orchestra.run.smithi076.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:30:38.171 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b: {} 2024-09-16T13:30:38.171 INFO:teuthology.orchestra.run.smithi076.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:30:38.179 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c: {} 2024-09-16T13:30:38.179 INFO:teuthology.orchestra.run.smithi076.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:30:38.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T13:30:39.113 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2024-09-16T13:30:39.114 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T13:30:39.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T13:30:40.119 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T13:30:40.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T13:30:41.119 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T13:30:41.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T13:30:42.129 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T13:30:42.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T13:30:43.139 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T13:30:43.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T13:30:44.147 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T13:30:44.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T13:30:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T13:30:44.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T13:30:46.166 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T13:30:46.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T13:30:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T13:30:47.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T13:30:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T13:30:48.191 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T13:30:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T13:30:49.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T13:30:50.191 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T13:30:50.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T13:30:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T13:30:51.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T13:30:52.198 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T13:30:52.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T13:30:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T13:30:53.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T13:30:54.211 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T13:30:54.227 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T13:30:54.227 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T13:30:54.227 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T13:30:54.227 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T13:30:54.227 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T13:30:54.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:30:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:54.498 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":166,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:30:54.203791+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:29:06.165279+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-16T13:28:16.900065+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"pool_client.0","create_time":"2024-09-16T13:28:25.583589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"4.b","osds":[6,4]},{"pgid":"4.d","osds":[5,0]}],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:30:54.510 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T13:30:54.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:30:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:54.770 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":166,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:30:54.203791+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:29:06.165279+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-16T13:28:16.900065+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"pool_client.0","create_time":"2024-09-16T13:28:25.583589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"4.b","osds":[6,4]},{"pgid":"4.d","osds":[5,0]}],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:30:54.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T13:30:54.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T13:30:54.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T13:30:54.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T13:30:54.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T13:30:54.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T13:30:54.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T13:30:54.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T13:30:55.016 INFO:teuthology.orchestra.run.smithi076.stdout:64424509479 2024-09-16T13:30:55.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:30:55.017 INFO:teuthology.orchestra.run.smithi076.stdout:60129542183 2024-09-16T13:30:55.018 INFO:teuthology.orchestra.run.smithi076.stdout:64424509480 2024-09-16T13:30:55.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:30:55.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:30:55.020 INFO:teuthology.orchestra.run.smithi076.stdout:60129542185 2024-09-16T13:30:55.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:30:55.021 INFO:teuthology.orchestra.run.smithi076.stdout:64424509479 2024-09-16T13:30:55.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:30:55.022 INFO:teuthology.orchestra.run.smithi076.stdout:60129542185 2024-09-16T13:30:55.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:30:55.024 INFO:teuthology.orchestra.run.smithi076.stdout:60129542183 2024-09-16T13:30:55.024 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:30:55.025 INFO:teuthology.orchestra.run.smithi076.stdout:64424509481 2024-09-16T13:30:55.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:30:55.385 INFO:teuthology.orchestra.run.smithi076.stdout:64424509477 2024-09-16T13:30:55.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509479 got 64424509477 for osd.3 2024-09-16T13:30:55.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T13:30:55.429+0000 7f0919465640 0 --2- 172.21.15.76:0/2509451318 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09140ab980 0x7f0914133b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T13:30:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:60129542184 2024-09-16T13:30:55.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.1 2024-09-16T13:30:55.555 INFO:teuthology.orchestra.run.smithi076.stdout:64424509477 2024-09-16T13:30:55.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509479 got 64424509477 for osd.7 2024-09-16T13:30:55.594 INFO:teuthology.orchestra.run.smithi076.stdout:60129542182 2024-09-16T13:30:55.601 INFO:teuthology.orchestra.run.smithi076.stdout:64424509478 2024-09-16T13:30:55.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542182 for osd.2 2024-09-16T13:30:55.615 INFO:teuthology.orchestra.run.smithi076.stdout:60129542182 2024-09-16T13:30:55.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509478 for osd.4 2024-09-16T13:30:55.623 INFO:teuthology.orchestra.run.smithi076.stdout:64424509479 2024-09-16T13:30:55.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542182 for osd.0 2024-09-16T13:30:55.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509479 for osd.5 2024-09-16T13:30:55.671 INFO:teuthology.orchestra.run.smithi076.stdout:60129542184 2024-09-16T13:30:55.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.6 2024-09-16T13:30:56.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:30:56.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:30:56.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:30:56.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:30:56.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:30:56.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:30:56.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:30:56.684 INFO:teuthology.orchestra.run.smithi076.stdout:64424509479 2024-09-16T13:30:56.685 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:30:56.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509479 got 64424509479 for osd.3 2024-09-16T13:30:56.704 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:56.847 INFO:teuthology.orchestra.run.smithi076.stdout:60129542185 2024-09-16T13:30:56.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542185 for osd.1 2024-09-16T13:30:56.867 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:56.958 INFO:teuthology.orchestra.run.smithi076.stdout:64424509479 2024-09-16T13:30:56.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509479 got 64424509479 for osd.7 2024-09-16T13:30:56.977 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.023 INFO:teuthology.orchestra.run.smithi076.stdout:64424509481 2024-09-16T13:30:57.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509481 for osd.5 2024-09-16T13:30:57.037 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.062 INFO:teuthology.orchestra.run.smithi076.stdout:64424509480 2024-09-16T13:30:57.071 INFO:teuthology.orchestra.run.smithi076.stdout:60129542185 2024-09-16T13:30:57.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.4 2024-09-16T13:30:57.075 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542185 for osd.6 2024-09-16T13:30:57.085 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.106 INFO:teuthology.orchestra.run.smithi076.stdout:60129542183 2024-09-16T13:30:57.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542183 for osd.2 2024-09-16T13:30:57.122 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.154 INFO:teuthology.orchestra.run.smithi076.stdout:60129542183 2024-09-16T13:30:57.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542183 got 60129542183 for osd.0 2024-09-16T13:30:57.166 DEBUG:teuthology.parallel:result is None 2024-09-16T13:30:57.166 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T13:30:57.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:57.469 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:57.470 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:57.481 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T13:30:55.868817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18527,"ondisk_log_size":18527,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436092,"kb_used_data":15620,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538628,"statfs":{"total":773094113280,"available":770599555072,"internally_reserved":0,"allocated":15994880,"data_stored":9272199,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.680758"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2264,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195747+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:52.195747+0000","last_peered":"2024-09-16T13:30:52.195747+0000","last_clean":"2024-09-16T13:30:52.195747+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:52.195747+0000","last_undegraded":"2024-09-16T13:30:52.195747+0000","last_fullsized":"2024-09-16T13:30:52.195747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11845,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012799+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:53.012799+0000","last_peered":"2024-09-16T13:30:53.012799+0000","last_clean":"2024-09-16T13:30:53.012799+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:53.012799+0000","last_undegraded":"2024-09-16T13:30:53.012799+0000","last_fullsized":"2024-09-16T13:30:53.012799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1680,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:52.197025+0000","last_peered":"2024-09-16T13:30:52.197025+0000","last_clean":"2024-09-16T13:30:52.197025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:52.197025+0000","last_undegraded":"2024-09-16T13:30:52.197025+0000","last_fullsized":"2024-09-16T13:30:52.197025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4601,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671843+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:53.671843+0000","last_peered":"2024-09-16T13:30:53.671843+0000","last_clean":"2024-09-16T13:30:53.671843+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:53.671843+0000","last_undegraded":"2024-09-16T13:30:53.671843+0000","last_fullsized":"2024-09-16T13:30:53.671843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2377,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374989+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:52.374989+0000","last_peered":"2024-09-16T13:30:52.374989+0000","last_clean":"2024-09-16T13:30:52.374989+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:52.374989+0000","last_undegraded":"2024-09-16T13:30:52.374989+0000","last_fullsized":"2024-09-16T13:30:52.374989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4334,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374920+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:52.374920+0000","last_peered":"2024-09-16T13:30:52.374920+0000","last_clean":"2024-09-16T13:30:52.374920+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:52.374920+0000","last_undegraded":"2024-09-16T13:30:52.374920+0000","last_fullsized":"2024-09-16T13:30:52.374920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3515,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197083+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:52.197083+0000","last_peered":"2024-09-16T13:30:52.197083+0000","last_clean":"2024-09-16T13:30:52.197083+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:52.197083+0000","last_undegraded":"2024-09-16T13:30:52.197083+0000","last_fullsized":"2024-09-16T13:30:52.197083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3577,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374915+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:52.374915+0000","last_peered":"2024-09-16T13:30:52.374915+0000","last_clean":"2024-09-16T13:30:52.374915+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:52.374915+0000","last_undegraded":"2024-09-16T13:30:52.374915+0000","last_fullsized":"2024-09-16T13:30:52.374915+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197135+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:52.197135+0000","last_peered":"2024-09-16T13:30:52.197135+0000","last_clean":"2024-09-16T13:30:52.197135+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:52.197135+0000","last_undegraded":"2024-09-16T13:30:52.197135+0000","last_fullsized":"2024-09-16T13:30:52.197135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2883,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012849+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:53.012849+0000","last_peered":"2024-09-16T13:30:53.012849+0000","last_clean":"2024-09-16T13:30:53.012849+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:53.012849+0000","last_undegraded":"2024-09-16T13:30:53.012849+0000","last_fullsized":"2024-09-16T13:30:53.012849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3513,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196998+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:52.196998+0000","last_peered":"2024-09-16T13:30:52.196998+0000","last_clean":"2024-09-16T13:30:52.196998+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:52.196998+0000","last_undegraded":"2024-09-16T13:30:52.196998+0000","last_fullsized":"2024-09-16T13:30:52.196998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2023,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375068+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:52.375068+0000","last_peered":"2024-09-16T13:30:52.375068+0000","last_clean":"2024-09-16T13:30:52.375068+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:52.375068+0000","last_undegraded":"2024-09-16T13:30:52.375068+0000","last_fullsized":"2024-09-16T13:30:52.375068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4363,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201629+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:53.201629+0000","last_peered":"2024-09-16T13:30:53.201629+0000","last_clean":"2024-09-16T13:30:53.201629+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:53.201629+0000","last_undegraded":"2024-09-16T13:30:53.201629+0000","last_fullsized":"2024-09-16T13:30:53.201629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4867,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196996+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:52.196996+0000","last_peered":"2024-09-16T13:30:52.196996+0000","last_clean":"2024-09-16T13:30:52.196996+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:52.196996+0000","last_undegraded":"2024-09-16T13:30:52.196996+0000","last_fullsized":"2024-09-16T13:30:52.196996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3938,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196714+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:52.196714+0000","last_peered":"2024-09-16T13:30:52.196714+0000","last_clean":"2024-09-16T13:30:52.196714+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:52.196714+0000","last_undegraded":"2024-09-16T13:30:52.196714+0000","last_fullsized":"2024-09-16T13:30:52.196714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195771+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:52.195771+0000","last_peered":"2024-09-16T13:30:52.195771+0000","last_clean":"2024-09-16T13:30:52.195771+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:52.195771+0000","last_undegraded":"2024-09-16T13:30:52.195771+0000","last_fullsized":"2024-09-16T13:30:52.195771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3621,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374947+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:52.374947+0000","last_peered":"2024-09-16T13:30:52.374947+0000","last_clean":"2024-09-16T13:30:52.374947+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:52.374947+0000","last_undegraded":"2024-09-16T13:30:52.374947+0000","last_fullsized":"2024-09-16T13:30:52.374947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201743+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201743+0000","last_peered":"2024-09-16T13:30:53.201743+0000","last_clean":"2024-09-16T13:30:53.201743+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:53.201743+0000","last_undegraded":"2024-09-16T13:30:53.201743+0000","last_fullsized":"2024-09-16T13:30:53.201743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2654,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196913+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:52.196913+0000","last_peered":"2024-09-16T13:30:52.196913+0000","last_clean":"2024-09-16T13:30:52.196913+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:52.196913+0000","last_undegraded":"2024-09-16T13:30:52.196913+0000","last_fullsized":"2024-09-16T13:30:52.196913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195591+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:52.195591+0000","last_peered":"2024-09-16T13:30:52.195591+0000","last_clean":"2024-09-16T13:30:52.195591+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:52.195591+0000","last_undegraded":"2024-09-16T13:30:52.195591+0000","last_fullsized":"2024-09-16T13:30:52.195591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012756+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:53.012756+0000","last_peered":"2024-09-16T13:30:53.012756+0000","last_clean":"2024-09-16T13:30:53.012756+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:53.012756+0000","last_undegraded":"2024-09-16T13:30:53.012756+0000","last_fullsized":"2024-09-16T13:30:53.012756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201795+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201795+0000","last_peered":"2024-09-16T13:30:53.201795+0000","last_clean":"2024-09-16T13:30:53.201795+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:53.201795+0000","last_undegraded":"2024-09-16T13:30:53.201795+0000","last_fullsized":"2024-09-16T13:30:53.201795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3497,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201742+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:53.201742+0000","last_peered":"2024-09-16T13:30:53.201742+0000","last_clean":"2024-09-16T13:30:53.201742+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:53.201742+0000","last_undegraded":"2024-09-16T13:30:53.201742+0000","last_fullsized":"2024-09-16T13:30:53.201742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375070+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:52.375070+0000","last_peered":"2024-09-16T13:30:52.375070+0000","last_clean":"2024-09-16T13:30:52.375070+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:52.375070+0000","last_undegraded":"2024-09-16T13:30:52.375070+0000","last_fullsized":"2024-09-16T13:30:52.375070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4566,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201694+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:53.201694+0000","last_peered":"2024-09-16T13:30:53.201694+0000","last_clean":"2024-09-16T13:30:53.201694+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:53.201694+0000","last_undegraded":"2024-09-16T13:30:53.201694+0000","last_fullsized":"2024-09-16T13:30:53.201694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201531+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:53.201531+0000","last_peered":"2024-09-16T13:30:53.201531+0000","last_clean":"2024-09-16T13:30:53.201531+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:53.201531+0000","last_undegraded":"2024-09-16T13:30:53.201531+0000","last_fullsized":"2024-09-16T13:30:53.201531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671758+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:53.671758+0000","last_peered":"2024-09-16T13:30:53.671758+0000","last_clean":"2024-09-16T13:30:53.671758+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:53.671758+0000","last_undegraded":"2024-09-16T13:30:53.671758+0000","last_fullsized":"2024-09-16T13:30:53.671758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3470,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201711+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:53.201711+0000","last_peered":"2024-09-16T13:30:53.201711+0000","last_clean":"2024-09-16T13:30:53.201711+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:53.201711+0000","last_undegraded":"2024-09-16T13:30:53.201711+0000","last_fullsized":"2024-09-16T13:30:53.201711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'3","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202251+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:53.202251+0000","last_peered":"2024-09-16T13:30:53.202251+0000","last_clean":"2024-09-16T13:30:53.202251+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:53.202251+0000","last_undegraded":"2024-09-16T13:30:53.202251+0000","last_fullsized":"2024-09-16T13:30:53.202251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013330+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.013330+0000","last_peered":"2024-09-16T13:30:53.013330+0000","last_clean":"2024-09-16T13:30:53.013330+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.013330+0000","last_undegraded":"2024-09-16T13:30:53.013330+0000","last_fullsized":"2024-09-16T13:30:53.013330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4081,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012913+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:53.012913+0000","last_peered":"2024-09-16T13:30:53.012913+0000","last_clean":"2024-09-16T13:30:53.012913+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:53.012913+0000","last_undegraded":"2024-09-16T13:30:53.012913+0000","last_fullsized":"2024-09-16T13:30:53.012913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":151,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671937+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.671937+0000","last_peered":"2024-09-16T13:30:53.671937+0000","last_clean":"2024-09-16T13:30:53.671937+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.671937+0000","last_undegraded":"2024-09-16T13:30:53.671937+0000","last_fullsized":"2024-09-16T13:30:53.671937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013268+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.013268+0000","last_peered":"2024-09-16T13:30:53.013268+0000","last_clean":"2024-09-16T13:30:53.013268+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.013268+0000","last_undegraded":"2024-09-16T13:30:53.013268+0000","last_fullsized":"2024-09-16T13:30:53.013268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2516,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195638+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:52.195638+0000","last_peered":"2024-09-16T13:30:52.195638+0000","last_clean":"2024-09-16T13:30:52.195638+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:52.195638+0000","last_undegraded":"2024-09-16T13:30:52.195638+0000","last_fullsized":"2024-09-16T13:30:52.195638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4818,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374909+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:52.374909+0000","last_peered":"2024-09-16T13:30:52.374909+0000","last_clean":"2024-09-16T13:30:52.374909+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:52.374909+0000","last_undegraded":"2024-09-16T13:30:52.374909+0000","last_fullsized":"2024-09-16T13:30:52.374909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4597,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195569+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:52.195569+0000","last_peered":"2024-09-16T13:30:52.195569+0000","last_clean":"2024-09-16T13:30:52.195569+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:52.195569+0000","last_undegraded":"2024-09-16T13:30:52.195569+0000","last_fullsized":"2024-09-16T13:30:52.195569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5510,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195686+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:52.195686+0000","last_peered":"2024-09-16T13:30:52.195686+0000","last_clean":"2024-09-16T13:30:52.195686+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:52.195686+0000","last_undegraded":"2024-09-16T13:30:52.195686+0000","last_fullsized":"2024-09-16T13:30:52.195686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5587,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375030+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:52.375030+0000","last_peered":"2024-09-16T13:30:52.375030+0000","last_clean":"2024-09-16T13:30:52.375030+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:52.375030+0000","last_undegraded":"2024-09-16T13:30:52.375030+0000","last_fullsized":"2024-09-16T13:30:52.375030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1804,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375050+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:52.375050+0000","last_peered":"2024-09-16T13:30:52.375050+0000","last_clean":"2024-09-16T13:30:52.375050+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:52.375050+0000","last_undegraded":"2024-09-16T13:30:52.375050+0000","last_fullsized":"2024-09-16T13:30:52.375050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671727+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:53.671727+0000","last_peered":"2024-09-16T13:30:53.671727+0000","last_clean":"2024-09-16T13:30:53.671727+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:53.671727+0000","last_undegraded":"2024-09-16T13:30:53.671727+0000","last_fullsized":"2024-09-16T13:30:53.671727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303492,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068348,"statfs":{"total":96636764160,"available":96325988352,"internally_reserved":0,"allocated":2019328,"data_stored":1600134,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.497}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.497}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.378},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1009951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.439},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.501},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52600000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509479,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.246},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.46999999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299292,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072548,"statfs":{"total":96636764160,"available":96330289152,"internally_reserved":0,"allocated":1978368,"data_stored":1009766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.502}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.376}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.48399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53000000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1982464,"data_stored":1009785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.378}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.56000000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299352,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072488,"statfs":{"total":96636764160,"available":96330227712,"internally_reserved":0,"allocated":2039808,"data_stored":1600217,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.438}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:57.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:57.747 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:57.748 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:57.759 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T13:30:55.868817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18527,"ondisk_log_size":18527,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436092,"kb_used_data":15620,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538628,"statfs":{"total":773094113280,"available":770599555072,"internally_reserved":0,"allocated":15994880,"data_stored":9272199,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.680758"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2264,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195747+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:52.195747+0000","last_peered":"2024-09-16T13:30:52.195747+0000","last_clean":"2024-09-16T13:30:52.195747+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:52.195747+0000","last_undegraded":"2024-09-16T13:30:52.195747+0000","last_fullsized":"2024-09-16T13:30:52.195747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11845,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012799+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:53.012799+0000","last_peered":"2024-09-16T13:30:53.012799+0000","last_clean":"2024-09-16T13:30:53.012799+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:53.012799+0000","last_undegraded":"2024-09-16T13:30:53.012799+0000","last_fullsized":"2024-09-16T13:30:53.012799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1680,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:52.197025+0000","last_peered":"2024-09-16T13:30:52.197025+0000","last_clean":"2024-09-16T13:30:52.197025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:52.197025+0000","last_undegraded":"2024-09-16T13:30:52.197025+0000","last_fullsized":"2024-09-16T13:30:52.197025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4601,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671843+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:53.671843+0000","last_peered":"2024-09-16T13:30:53.671843+0000","last_clean":"2024-09-16T13:30:53.671843+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:53.671843+0000","last_undegraded":"2024-09-16T13:30:53.671843+0000","last_fullsized":"2024-09-16T13:30:53.671843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2377,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374989+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:52.374989+0000","last_peered":"2024-09-16T13:30:52.374989+0000","last_clean":"2024-09-16T13:30:52.374989+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:52.374989+0000","last_undegraded":"2024-09-16T13:30:52.374989+0000","last_fullsized":"2024-09-16T13:30:52.374989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4334,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374920+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:52.374920+0000","last_peered":"2024-09-16T13:30:52.374920+0000","last_clean":"2024-09-16T13:30:52.374920+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:52.374920+0000","last_undegraded":"2024-09-16T13:30:52.374920+0000","last_fullsized":"2024-09-16T13:30:52.374920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3515,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197083+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:52.197083+0000","last_peered":"2024-09-16T13:30:52.197083+0000","last_clean":"2024-09-16T13:30:52.197083+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:52.197083+0000","last_undegraded":"2024-09-16T13:30:52.197083+0000","last_fullsized":"2024-09-16T13:30:52.197083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3577,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374915+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:52.374915+0000","last_peered":"2024-09-16T13:30:52.374915+0000","last_clean":"2024-09-16T13:30:52.374915+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:52.374915+0000","last_undegraded":"2024-09-16T13:30:52.374915+0000","last_fullsized":"2024-09-16T13:30:52.374915+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197135+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:52.197135+0000","last_peered":"2024-09-16T13:30:52.197135+0000","last_clean":"2024-09-16T13:30:52.197135+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:52.197135+0000","last_undegraded":"2024-09-16T13:30:52.197135+0000","last_fullsized":"2024-09-16T13:30:52.197135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2883,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012849+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:53.012849+0000","last_peered":"2024-09-16T13:30:53.012849+0000","last_clean":"2024-09-16T13:30:53.012849+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:53.012849+0000","last_undegraded":"2024-09-16T13:30:53.012849+0000","last_fullsized":"2024-09-16T13:30:53.012849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3513,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196998+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:52.196998+0000","last_peered":"2024-09-16T13:30:52.196998+0000","last_clean":"2024-09-16T13:30:52.196998+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:52.196998+0000","last_undegraded":"2024-09-16T13:30:52.196998+0000","last_fullsized":"2024-09-16T13:30:52.196998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2023,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375068+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:52.375068+0000","last_peered":"2024-09-16T13:30:52.375068+0000","last_clean":"2024-09-16T13:30:52.375068+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:52.375068+0000","last_undegraded":"2024-09-16T13:30:52.375068+0000","last_fullsized":"2024-09-16T13:30:52.375068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4363,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201629+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:53.201629+0000","last_peered":"2024-09-16T13:30:53.201629+0000","last_clean":"2024-09-16T13:30:53.201629+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:53.201629+0000","last_undegraded":"2024-09-16T13:30:53.201629+0000","last_fullsized":"2024-09-16T13:30:53.201629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4867,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196996+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:52.196996+0000","last_peered":"2024-09-16T13:30:52.196996+0000","last_clean":"2024-09-16T13:30:52.196996+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:52.196996+0000","last_undegraded":"2024-09-16T13:30:52.196996+0000","last_fullsized":"2024-09-16T13:30:52.196996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3938,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196714+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:52.196714+0000","last_peered":"2024-09-16T13:30:52.196714+0000","last_clean":"2024-09-16T13:30:52.196714+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:52.196714+0000","last_undegraded":"2024-09-16T13:30:52.196714+0000","last_fullsized":"2024-09-16T13:30:52.196714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195771+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:52.195771+0000","last_peered":"2024-09-16T13:30:52.195771+0000","last_clean":"2024-09-16T13:30:52.195771+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:52.195771+0000","last_undegraded":"2024-09-16T13:30:52.195771+0000","last_fullsized":"2024-09-16T13:30:52.195771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3621,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374947+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:52.374947+0000","last_peered":"2024-09-16T13:30:52.374947+0000","last_clean":"2024-09-16T13:30:52.374947+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:52.374947+0000","last_undegraded":"2024-09-16T13:30:52.374947+0000","last_fullsized":"2024-09-16T13:30:52.374947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201743+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201743+0000","last_peered":"2024-09-16T13:30:53.201743+0000","last_clean":"2024-09-16T13:30:53.201743+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:53.201743+0000","last_undegraded":"2024-09-16T13:30:53.201743+0000","last_fullsized":"2024-09-16T13:30:53.201743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2654,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196913+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:52.196913+0000","last_peered":"2024-09-16T13:30:52.196913+0000","last_clean":"2024-09-16T13:30:52.196913+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:52.196913+0000","last_undegraded":"2024-09-16T13:30:52.196913+0000","last_fullsized":"2024-09-16T13:30:52.196913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195591+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:52.195591+0000","last_peered":"2024-09-16T13:30:52.195591+0000","last_clean":"2024-09-16T13:30:52.195591+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:52.195591+0000","last_undegraded":"2024-09-16T13:30:52.195591+0000","last_fullsized":"2024-09-16T13:30:52.195591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012756+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:53.012756+0000","last_peered":"2024-09-16T13:30:53.012756+0000","last_clean":"2024-09-16T13:30:53.012756+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:53.012756+0000","last_undegraded":"2024-09-16T13:30:53.012756+0000","last_fullsized":"2024-09-16T13:30:53.012756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201795+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201795+0000","last_peered":"2024-09-16T13:30:53.201795+0000","last_clean":"2024-09-16T13:30:53.201795+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:53.201795+0000","last_undegraded":"2024-09-16T13:30:53.201795+0000","last_fullsized":"2024-09-16T13:30:53.201795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3497,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201742+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:53.201742+0000","last_peered":"2024-09-16T13:30:53.201742+0000","last_clean":"2024-09-16T13:30:53.201742+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:53.201742+0000","last_undegraded":"2024-09-16T13:30:53.201742+0000","last_fullsized":"2024-09-16T13:30:53.201742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375070+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:52.375070+0000","last_peered":"2024-09-16T13:30:52.375070+0000","last_clean":"2024-09-16T13:30:52.375070+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:52.375070+0000","last_undegraded":"2024-09-16T13:30:52.375070+0000","last_fullsized":"2024-09-16T13:30:52.375070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4566,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201694+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:53.201694+0000","last_peered":"2024-09-16T13:30:53.201694+0000","last_clean":"2024-09-16T13:30:53.201694+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:53.201694+0000","last_undegraded":"2024-09-16T13:30:53.201694+0000","last_fullsized":"2024-09-16T13:30:53.201694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201531+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:53.201531+0000","last_peered":"2024-09-16T13:30:53.201531+0000","last_clean":"2024-09-16T13:30:53.201531+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:53.201531+0000","last_undegraded":"2024-09-16T13:30:53.201531+0000","last_fullsized":"2024-09-16T13:30:53.201531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671758+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:53.671758+0000","last_peered":"2024-09-16T13:30:53.671758+0000","last_clean":"2024-09-16T13:30:53.671758+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:53.671758+0000","last_undegraded":"2024-09-16T13:30:53.671758+0000","last_fullsized":"2024-09-16T13:30:53.671758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3470,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201711+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:53.201711+0000","last_peered":"2024-09-16T13:30:53.201711+0000","last_clean":"2024-09-16T13:30:53.201711+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:53.201711+0000","last_undegraded":"2024-09-16T13:30:53.201711+0000","last_fullsized":"2024-09-16T13:30:53.201711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'3","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202251+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:53.202251+0000","last_peered":"2024-09-16T13:30:53.202251+0000","last_clean":"2024-09-16T13:30:53.202251+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:53.202251+0000","last_undegraded":"2024-09-16T13:30:53.202251+0000","last_fullsized":"2024-09-16T13:30:53.202251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013330+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.013330+0000","last_peered":"2024-09-16T13:30:53.013330+0000","last_clean":"2024-09-16T13:30:53.013330+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.013330+0000","last_undegraded":"2024-09-16T13:30:53.013330+0000","last_fullsized":"2024-09-16T13:30:53.013330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4081,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012913+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:53.012913+0000","last_peered":"2024-09-16T13:30:53.012913+0000","last_clean":"2024-09-16T13:30:53.012913+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:53.012913+0000","last_undegraded":"2024-09-16T13:30:53.012913+0000","last_fullsized":"2024-09-16T13:30:53.012913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":151,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671937+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.671937+0000","last_peered":"2024-09-16T13:30:53.671937+0000","last_clean":"2024-09-16T13:30:53.671937+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.671937+0000","last_undegraded":"2024-09-16T13:30:53.671937+0000","last_fullsized":"2024-09-16T13:30:53.671937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013268+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.013268+0000","last_peered":"2024-09-16T13:30:53.013268+0000","last_clean":"2024-09-16T13:30:53.013268+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.013268+0000","last_undegraded":"2024-09-16T13:30:53.013268+0000","last_fullsized":"2024-09-16T13:30:53.013268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2516,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195638+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:52.195638+0000","last_peered":"2024-09-16T13:30:52.195638+0000","last_clean":"2024-09-16T13:30:52.195638+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:52.195638+0000","last_undegraded":"2024-09-16T13:30:52.195638+0000","last_fullsized":"2024-09-16T13:30:52.195638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4818,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374909+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:52.374909+0000","last_peered":"2024-09-16T13:30:52.374909+0000","last_clean":"2024-09-16T13:30:52.374909+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:52.374909+0000","last_undegraded":"2024-09-16T13:30:52.374909+0000","last_fullsized":"2024-09-16T13:30:52.374909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4597,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195569+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:52.195569+0000","last_peered":"2024-09-16T13:30:52.195569+0000","last_clean":"2024-09-16T13:30:52.195569+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:52.195569+0000","last_undegraded":"2024-09-16T13:30:52.195569+0000","last_fullsized":"2024-09-16T13:30:52.195569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5510,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195686+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:52.195686+0000","last_peered":"2024-09-16T13:30:52.195686+0000","last_clean":"2024-09-16T13:30:52.195686+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:52.195686+0000","last_undegraded":"2024-09-16T13:30:52.195686+0000","last_fullsized":"2024-09-16T13:30:52.195686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5587,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375030+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:52.375030+0000","last_peered":"2024-09-16T13:30:52.375030+0000","last_clean":"2024-09-16T13:30:52.375030+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:52.375030+0000","last_undegraded":"2024-09-16T13:30:52.375030+0000","last_fullsized":"2024-09-16T13:30:52.375030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1804,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375050+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:52.375050+0000","last_peered":"2024-09-16T13:30:52.375050+0000","last_clean":"2024-09-16T13:30:52.375050+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:52.375050+0000","last_undegraded":"2024-09-16T13:30:52.375050+0000","last_fullsized":"2024-09-16T13:30:52.375050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671727+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:53.671727+0000","last_peered":"2024-09-16T13:30:53.671727+0000","last_clean":"2024-09-16T13:30:53.671727+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:53.671727+0000","last_undegraded":"2024-09-16T13:30:53.671727+0000","last_fullsized":"2024-09-16T13:30:53.671727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303492,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068348,"statfs":{"total":96636764160,"available":96325988352,"internally_reserved":0,"allocated":2019328,"data_stored":1600134,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.497}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.497}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.378},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1009951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.439},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.501},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52600000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509479,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.246},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.46999999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299292,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072548,"statfs":{"total":96636764160,"available":96330289152,"internally_reserved":0,"allocated":1978368,"data_stored":1009766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.502}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.376}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.48399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53000000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1982464,"data_stored":1009785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.378}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.56000000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299352,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072488,"statfs":{"total":96636764160,"available":96330227712,"internally_reserved":0,"allocated":2039808,"data_stored":1600217,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.438}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:57.762 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T13:30:57.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T13:30:58.043 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:30:58.056 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-16T13:30:58.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:30:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:58.318 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":166,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:30:54.203791+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:29:06.165279+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-16T13:28:16.900065+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"pool_client.0","create_time":"2024-09-16T13:28:25.583589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"4.b","osds":[6,4]},{"pgid":"4.d","osds":[5,0]}],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:30:58.331 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:30:58.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:58.592 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:58.603 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-16T13:30:57.869183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18527,"ondisk_log_size":18527,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436092,"kb_used_data":15620,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538628,"statfs":{"total":773094113280,"available":770599555072,"internally_reserved":0,"allocated":15994880,"data_stored":9272199,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000939"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2264,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195747+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:52.195747+0000","last_peered":"2024-09-16T13:30:52.195747+0000","last_clean":"2024-09-16T13:30:52.195747+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:52.195747+0000","last_undegraded":"2024-09-16T13:30:52.195747+0000","last_fullsized":"2024-09-16T13:30:52.195747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11845,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012799+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:53.012799+0000","last_peered":"2024-09-16T13:30:53.012799+0000","last_clean":"2024-09-16T13:30:53.012799+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:53.012799+0000","last_undegraded":"2024-09-16T13:30:53.012799+0000","last_fullsized":"2024-09-16T13:30:53.012799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1680,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:52.197025+0000","last_peered":"2024-09-16T13:30:52.197025+0000","last_clean":"2024-09-16T13:30:52.197025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:52.197025+0000","last_undegraded":"2024-09-16T13:30:52.197025+0000","last_fullsized":"2024-09-16T13:30:52.197025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4601,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671843+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:53.671843+0000","last_peered":"2024-09-16T13:30:53.671843+0000","last_clean":"2024-09-16T13:30:53.671843+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:53.671843+0000","last_undegraded":"2024-09-16T13:30:53.671843+0000","last_fullsized":"2024-09-16T13:30:53.671843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2377,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374989+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:52.374989+0000","last_peered":"2024-09-16T13:30:52.374989+0000","last_clean":"2024-09-16T13:30:52.374989+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:52.374989+0000","last_undegraded":"2024-09-16T13:30:52.374989+0000","last_fullsized":"2024-09-16T13:30:52.374989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4334,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374920+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:52.374920+0000","last_peered":"2024-09-16T13:30:52.374920+0000","last_clean":"2024-09-16T13:30:52.374920+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:52.374920+0000","last_undegraded":"2024-09-16T13:30:52.374920+0000","last_fullsized":"2024-09-16T13:30:52.374920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3515,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197083+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:52.197083+0000","last_peered":"2024-09-16T13:30:52.197083+0000","last_clean":"2024-09-16T13:30:52.197083+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:52.197083+0000","last_undegraded":"2024-09-16T13:30:52.197083+0000","last_fullsized":"2024-09-16T13:30:52.197083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3577,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374915+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:52.374915+0000","last_peered":"2024-09-16T13:30:52.374915+0000","last_clean":"2024-09-16T13:30:52.374915+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:52.374915+0000","last_undegraded":"2024-09-16T13:30:52.374915+0000","last_fullsized":"2024-09-16T13:30:52.374915+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197135+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:52.197135+0000","last_peered":"2024-09-16T13:30:52.197135+0000","last_clean":"2024-09-16T13:30:52.197135+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:52.197135+0000","last_undegraded":"2024-09-16T13:30:52.197135+0000","last_fullsized":"2024-09-16T13:30:52.197135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2883,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012849+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:53.012849+0000","last_peered":"2024-09-16T13:30:53.012849+0000","last_clean":"2024-09-16T13:30:53.012849+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:53.012849+0000","last_undegraded":"2024-09-16T13:30:53.012849+0000","last_fullsized":"2024-09-16T13:30:53.012849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3513,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196998+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:52.196998+0000","last_peered":"2024-09-16T13:30:52.196998+0000","last_clean":"2024-09-16T13:30:52.196998+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:52.196998+0000","last_undegraded":"2024-09-16T13:30:52.196998+0000","last_fullsized":"2024-09-16T13:30:52.196998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2023,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375068+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:52.375068+0000","last_peered":"2024-09-16T13:30:52.375068+0000","last_clean":"2024-09-16T13:30:52.375068+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:52.375068+0000","last_undegraded":"2024-09-16T13:30:52.375068+0000","last_fullsized":"2024-09-16T13:30:52.375068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4363,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201629+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:53.201629+0000","last_peered":"2024-09-16T13:30:53.201629+0000","last_clean":"2024-09-16T13:30:53.201629+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:53.201629+0000","last_undegraded":"2024-09-16T13:30:53.201629+0000","last_fullsized":"2024-09-16T13:30:53.201629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4867,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196996+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:52.196996+0000","last_peered":"2024-09-16T13:30:52.196996+0000","last_clean":"2024-09-16T13:30:52.196996+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:52.196996+0000","last_undegraded":"2024-09-16T13:30:52.196996+0000","last_fullsized":"2024-09-16T13:30:52.196996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3938,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196714+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:52.196714+0000","last_peered":"2024-09-16T13:30:52.196714+0000","last_clean":"2024-09-16T13:30:52.196714+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:52.196714+0000","last_undegraded":"2024-09-16T13:30:52.196714+0000","last_fullsized":"2024-09-16T13:30:52.196714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195771+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:52.195771+0000","last_peered":"2024-09-16T13:30:52.195771+0000","last_clean":"2024-09-16T13:30:52.195771+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:52.195771+0000","last_undegraded":"2024-09-16T13:30:52.195771+0000","last_fullsized":"2024-09-16T13:30:52.195771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3621,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374947+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:52.374947+0000","last_peered":"2024-09-16T13:30:52.374947+0000","last_clean":"2024-09-16T13:30:52.374947+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:52.374947+0000","last_undegraded":"2024-09-16T13:30:52.374947+0000","last_fullsized":"2024-09-16T13:30:52.374947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201743+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201743+0000","last_peered":"2024-09-16T13:30:53.201743+0000","last_clean":"2024-09-16T13:30:53.201743+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:53.201743+0000","last_undegraded":"2024-09-16T13:30:53.201743+0000","last_fullsized":"2024-09-16T13:30:53.201743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2654,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196913+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:52.196913+0000","last_peered":"2024-09-16T13:30:52.196913+0000","last_clean":"2024-09-16T13:30:52.196913+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:52.196913+0000","last_undegraded":"2024-09-16T13:30:52.196913+0000","last_fullsized":"2024-09-16T13:30:52.196913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195591+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:52.195591+0000","last_peered":"2024-09-16T13:30:52.195591+0000","last_clean":"2024-09-16T13:30:52.195591+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:52.195591+0000","last_undegraded":"2024-09-16T13:30:52.195591+0000","last_fullsized":"2024-09-16T13:30:52.195591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012756+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:53.012756+0000","last_peered":"2024-09-16T13:30:53.012756+0000","last_clean":"2024-09-16T13:30:53.012756+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:53.012756+0000","last_undegraded":"2024-09-16T13:30:53.012756+0000","last_fullsized":"2024-09-16T13:30:53.012756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201795+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201795+0000","last_peered":"2024-09-16T13:30:53.201795+0000","last_clean":"2024-09-16T13:30:53.201795+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:53.201795+0000","last_undegraded":"2024-09-16T13:30:53.201795+0000","last_fullsized":"2024-09-16T13:30:53.201795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3497,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201742+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:53.201742+0000","last_peered":"2024-09-16T13:30:53.201742+0000","last_clean":"2024-09-16T13:30:53.201742+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:53.201742+0000","last_undegraded":"2024-09-16T13:30:53.201742+0000","last_fullsized":"2024-09-16T13:30:53.201742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375070+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:52.375070+0000","last_peered":"2024-09-16T13:30:52.375070+0000","last_clean":"2024-09-16T13:30:52.375070+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:52.375070+0000","last_undegraded":"2024-09-16T13:30:52.375070+0000","last_fullsized":"2024-09-16T13:30:52.375070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4566,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201694+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:53.201694+0000","last_peered":"2024-09-16T13:30:53.201694+0000","last_clean":"2024-09-16T13:30:53.201694+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:53.201694+0000","last_undegraded":"2024-09-16T13:30:53.201694+0000","last_fullsized":"2024-09-16T13:30:53.201694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201531+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:53.201531+0000","last_peered":"2024-09-16T13:30:53.201531+0000","last_clean":"2024-09-16T13:30:53.201531+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:53.201531+0000","last_undegraded":"2024-09-16T13:30:53.201531+0000","last_fullsized":"2024-09-16T13:30:53.201531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671758+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:53.671758+0000","last_peered":"2024-09-16T13:30:53.671758+0000","last_clean":"2024-09-16T13:30:53.671758+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:53.671758+0000","last_undegraded":"2024-09-16T13:30:53.671758+0000","last_fullsized":"2024-09-16T13:30:53.671758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3470,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201711+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:53.201711+0000","last_peered":"2024-09-16T13:30:53.201711+0000","last_clean":"2024-09-16T13:30:53.201711+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:53.201711+0000","last_undegraded":"2024-09-16T13:30:53.201711+0000","last_fullsized":"2024-09-16T13:30:53.201711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'3","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202251+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:53.202251+0000","last_peered":"2024-09-16T13:30:53.202251+0000","last_clean":"2024-09-16T13:30:53.202251+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:53.202251+0000","last_undegraded":"2024-09-16T13:30:53.202251+0000","last_fullsized":"2024-09-16T13:30:53.202251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013330+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.013330+0000","last_peered":"2024-09-16T13:30:53.013330+0000","last_clean":"2024-09-16T13:30:53.013330+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.013330+0000","last_undegraded":"2024-09-16T13:30:53.013330+0000","last_fullsized":"2024-09-16T13:30:53.013330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4081,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012913+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:53.012913+0000","last_peered":"2024-09-16T13:30:53.012913+0000","last_clean":"2024-09-16T13:30:53.012913+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:53.012913+0000","last_undegraded":"2024-09-16T13:30:53.012913+0000","last_fullsized":"2024-09-16T13:30:53.012913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":151,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671937+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.671937+0000","last_peered":"2024-09-16T13:30:53.671937+0000","last_clean":"2024-09-16T13:30:53.671937+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.671937+0000","last_undegraded":"2024-09-16T13:30:53.671937+0000","last_fullsized":"2024-09-16T13:30:53.671937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013268+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.013268+0000","last_peered":"2024-09-16T13:30:53.013268+0000","last_clean":"2024-09-16T13:30:53.013268+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.013268+0000","last_undegraded":"2024-09-16T13:30:53.013268+0000","last_fullsized":"2024-09-16T13:30:53.013268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2516,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195638+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:52.195638+0000","last_peered":"2024-09-16T13:30:52.195638+0000","last_clean":"2024-09-16T13:30:52.195638+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:52.195638+0000","last_undegraded":"2024-09-16T13:30:52.195638+0000","last_fullsized":"2024-09-16T13:30:52.195638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4818,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374909+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:52.374909+0000","last_peered":"2024-09-16T13:30:52.374909+0000","last_clean":"2024-09-16T13:30:52.374909+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:52.374909+0000","last_undegraded":"2024-09-16T13:30:52.374909+0000","last_fullsized":"2024-09-16T13:30:52.374909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4597,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195569+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:52.195569+0000","last_peered":"2024-09-16T13:30:52.195569+0000","last_clean":"2024-09-16T13:30:52.195569+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:52.195569+0000","last_undegraded":"2024-09-16T13:30:52.195569+0000","last_fullsized":"2024-09-16T13:30:52.195569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5510,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195686+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:52.195686+0000","last_peered":"2024-09-16T13:30:52.195686+0000","last_clean":"2024-09-16T13:30:52.195686+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:52.195686+0000","last_undegraded":"2024-09-16T13:30:52.195686+0000","last_fullsized":"2024-09-16T13:30:52.195686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5587,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375030+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:52.375030+0000","last_peered":"2024-09-16T13:30:52.375030+0000","last_clean":"2024-09-16T13:30:52.375030+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:52.375030+0000","last_undegraded":"2024-09-16T13:30:52.375030+0000","last_fullsized":"2024-09-16T13:30:52.375030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1804,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375050+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:52.375050+0000","last_peered":"2024-09-16T13:30:52.375050+0000","last_clean":"2024-09-16T13:30:52.375050+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:52.375050+0000","last_undegraded":"2024-09-16T13:30:52.375050+0000","last_fullsized":"2024-09-16T13:30:52.375050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671727+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:53.671727+0000","last_peered":"2024-09-16T13:30:53.671727+0000","last_clean":"2024-09-16T13:30:53.671727+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:53.671727+0000","last_undegraded":"2024-09-16T13:30:53.671727+0000","last_fullsized":"2024-09-16T13:30:53.671727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303492,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068348,"statfs":{"total":96636764160,"available":96325988352,"internally_reserved":0,"allocated":2019328,"data_stored":1600134,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.497}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.497}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.378},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1009951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.439},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.501},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52600000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509479,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.246},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.46999999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299292,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072548,"statfs":{"total":96636764160,"available":96330289152,"internally_reserved":0,"allocated":1978368,"data_stored":1009766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.502}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.376}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.48399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53000000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1982464,"data_stored":1009785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.378}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.56000000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299352,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072488,"statfs":{"total":96636764160,"available":96330227712,"internally_reserved":0,"allocated":2039808,"data_stored":1600217,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.438}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:58.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:58.866 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:58.876 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-16T13:30:57.869183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18527,"ondisk_log_size":18527,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436092,"kb_used_data":15620,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538628,"statfs":{"total":773094113280,"available":770599555072,"internally_reserved":0,"allocated":15994880,"data_stored":9272199,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000939"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2264,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195747+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:52.195747+0000","last_peered":"2024-09-16T13:30:52.195747+0000","last_clean":"2024-09-16T13:30:52.195747+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:52.195747+0000","last_undegraded":"2024-09-16T13:30:52.195747+0000","last_fullsized":"2024-09-16T13:30:52.195747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11845,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012799+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:53.012799+0000","last_peered":"2024-09-16T13:30:53.012799+0000","last_clean":"2024-09-16T13:30:53.012799+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:53.012799+0000","last_undegraded":"2024-09-16T13:30:53.012799+0000","last_fullsized":"2024-09-16T13:30:53.012799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1680,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:52.197025+0000","last_peered":"2024-09-16T13:30:52.197025+0000","last_clean":"2024-09-16T13:30:52.197025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:52.197025+0000","last_undegraded":"2024-09-16T13:30:52.197025+0000","last_fullsized":"2024-09-16T13:30:52.197025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4601,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671843+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:53.671843+0000","last_peered":"2024-09-16T13:30:53.671843+0000","last_clean":"2024-09-16T13:30:53.671843+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:53.671843+0000","last_undegraded":"2024-09-16T13:30:53.671843+0000","last_fullsized":"2024-09-16T13:30:53.671843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2377,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374989+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:52.374989+0000","last_peered":"2024-09-16T13:30:52.374989+0000","last_clean":"2024-09-16T13:30:52.374989+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:52.374989+0000","last_undegraded":"2024-09-16T13:30:52.374989+0000","last_fullsized":"2024-09-16T13:30:52.374989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4334,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374920+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:52.374920+0000","last_peered":"2024-09-16T13:30:52.374920+0000","last_clean":"2024-09-16T13:30:52.374920+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:52.374920+0000","last_undegraded":"2024-09-16T13:30:52.374920+0000","last_fullsized":"2024-09-16T13:30:52.374920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3515,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197083+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:52.197083+0000","last_peered":"2024-09-16T13:30:52.197083+0000","last_clean":"2024-09-16T13:30:52.197083+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:52.197083+0000","last_undegraded":"2024-09-16T13:30:52.197083+0000","last_fullsized":"2024-09-16T13:30:52.197083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3577,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374915+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:52.374915+0000","last_peered":"2024-09-16T13:30:52.374915+0000","last_clean":"2024-09-16T13:30:52.374915+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:52.374915+0000","last_undegraded":"2024-09-16T13:30:52.374915+0000","last_fullsized":"2024-09-16T13:30:52.374915+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197135+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:52.197135+0000","last_peered":"2024-09-16T13:30:52.197135+0000","last_clean":"2024-09-16T13:30:52.197135+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:52.197135+0000","last_undegraded":"2024-09-16T13:30:52.197135+0000","last_fullsized":"2024-09-16T13:30:52.197135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2883,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012849+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:53.012849+0000","last_peered":"2024-09-16T13:30:53.012849+0000","last_clean":"2024-09-16T13:30:53.012849+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:53.012849+0000","last_undegraded":"2024-09-16T13:30:53.012849+0000","last_fullsized":"2024-09-16T13:30:53.012849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3513,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196998+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:52.196998+0000","last_peered":"2024-09-16T13:30:52.196998+0000","last_clean":"2024-09-16T13:30:52.196998+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:52.196998+0000","last_undegraded":"2024-09-16T13:30:52.196998+0000","last_fullsized":"2024-09-16T13:30:52.196998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2023,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375068+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:52.375068+0000","last_peered":"2024-09-16T13:30:52.375068+0000","last_clean":"2024-09-16T13:30:52.375068+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:52.375068+0000","last_undegraded":"2024-09-16T13:30:52.375068+0000","last_fullsized":"2024-09-16T13:30:52.375068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4363,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201629+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:53.201629+0000","last_peered":"2024-09-16T13:30:53.201629+0000","last_clean":"2024-09-16T13:30:53.201629+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:53.201629+0000","last_undegraded":"2024-09-16T13:30:53.201629+0000","last_fullsized":"2024-09-16T13:30:53.201629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4867,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196996+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:52.196996+0000","last_peered":"2024-09-16T13:30:52.196996+0000","last_clean":"2024-09-16T13:30:52.196996+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:52.196996+0000","last_undegraded":"2024-09-16T13:30:52.196996+0000","last_fullsized":"2024-09-16T13:30:52.196996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3938,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196714+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:52.196714+0000","last_peered":"2024-09-16T13:30:52.196714+0000","last_clean":"2024-09-16T13:30:52.196714+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:52.196714+0000","last_undegraded":"2024-09-16T13:30:52.196714+0000","last_fullsized":"2024-09-16T13:30:52.196714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195771+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:52.195771+0000","last_peered":"2024-09-16T13:30:52.195771+0000","last_clean":"2024-09-16T13:30:52.195771+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:52.195771+0000","last_undegraded":"2024-09-16T13:30:52.195771+0000","last_fullsized":"2024-09-16T13:30:52.195771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3621,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374947+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:52.374947+0000","last_peered":"2024-09-16T13:30:52.374947+0000","last_clean":"2024-09-16T13:30:52.374947+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:52.374947+0000","last_undegraded":"2024-09-16T13:30:52.374947+0000","last_fullsized":"2024-09-16T13:30:52.374947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201743+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201743+0000","last_peered":"2024-09-16T13:30:53.201743+0000","last_clean":"2024-09-16T13:30:53.201743+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:53.201743+0000","last_undegraded":"2024-09-16T13:30:53.201743+0000","last_fullsized":"2024-09-16T13:30:53.201743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2654,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196913+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:52.196913+0000","last_peered":"2024-09-16T13:30:52.196913+0000","last_clean":"2024-09-16T13:30:52.196913+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:52.196913+0000","last_undegraded":"2024-09-16T13:30:52.196913+0000","last_fullsized":"2024-09-16T13:30:52.196913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195591+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:52.195591+0000","last_peered":"2024-09-16T13:30:52.195591+0000","last_clean":"2024-09-16T13:30:52.195591+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:52.195591+0000","last_undegraded":"2024-09-16T13:30:52.195591+0000","last_fullsized":"2024-09-16T13:30:52.195591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012756+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:53.012756+0000","last_peered":"2024-09-16T13:30:53.012756+0000","last_clean":"2024-09-16T13:30:53.012756+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:53.012756+0000","last_undegraded":"2024-09-16T13:30:53.012756+0000","last_fullsized":"2024-09-16T13:30:53.012756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201795+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201795+0000","last_peered":"2024-09-16T13:30:53.201795+0000","last_clean":"2024-09-16T13:30:53.201795+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:53.201795+0000","last_undegraded":"2024-09-16T13:30:53.201795+0000","last_fullsized":"2024-09-16T13:30:53.201795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3497,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201742+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:53.201742+0000","last_peered":"2024-09-16T13:30:53.201742+0000","last_clean":"2024-09-16T13:30:53.201742+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:53.201742+0000","last_undegraded":"2024-09-16T13:30:53.201742+0000","last_fullsized":"2024-09-16T13:30:53.201742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375070+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:52.375070+0000","last_peered":"2024-09-16T13:30:52.375070+0000","last_clean":"2024-09-16T13:30:52.375070+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:52.375070+0000","last_undegraded":"2024-09-16T13:30:52.375070+0000","last_fullsized":"2024-09-16T13:30:52.375070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4566,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201694+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:53.201694+0000","last_peered":"2024-09-16T13:30:53.201694+0000","last_clean":"2024-09-16T13:30:53.201694+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:53.201694+0000","last_undegraded":"2024-09-16T13:30:53.201694+0000","last_fullsized":"2024-09-16T13:30:53.201694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201531+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:53.201531+0000","last_peered":"2024-09-16T13:30:53.201531+0000","last_clean":"2024-09-16T13:30:53.201531+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:53.201531+0000","last_undegraded":"2024-09-16T13:30:53.201531+0000","last_fullsized":"2024-09-16T13:30:53.201531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671758+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:53.671758+0000","last_peered":"2024-09-16T13:30:53.671758+0000","last_clean":"2024-09-16T13:30:53.671758+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:53.671758+0000","last_undegraded":"2024-09-16T13:30:53.671758+0000","last_fullsized":"2024-09-16T13:30:53.671758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3470,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201711+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:53.201711+0000","last_peered":"2024-09-16T13:30:53.201711+0000","last_clean":"2024-09-16T13:30:53.201711+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:53.201711+0000","last_undegraded":"2024-09-16T13:30:53.201711+0000","last_fullsized":"2024-09-16T13:30:53.201711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'3","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202251+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:53.202251+0000","last_peered":"2024-09-16T13:30:53.202251+0000","last_clean":"2024-09-16T13:30:53.202251+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:53.202251+0000","last_undegraded":"2024-09-16T13:30:53.202251+0000","last_fullsized":"2024-09-16T13:30:53.202251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013330+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.013330+0000","last_peered":"2024-09-16T13:30:53.013330+0000","last_clean":"2024-09-16T13:30:53.013330+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.013330+0000","last_undegraded":"2024-09-16T13:30:53.013330+0000","last_fullsized":"2024-09-16T13:30:53.013330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4081,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012913+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:53.012913+0000","last_peered":"2024-09-16T13:30:53.012913+0000","last_clean":"2024-09-16T13:30:53.012913+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:53.012913+0000","last_undegraded":"2024-09-16T13:30:53.012913+0000","last_fullsized":"2024-09-16T13:30:53.012913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":151,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671937+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.671937+0000","last_peered":"2024-09-16T13:30:53.671937+0000","last_clean":"2024-09-16T13:30:53.671937+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.671937+0000","last_undegraded":"2024-09-16T13:30:53.671937+0000","last_fullsized":"2024-09-16T13:30:53.671937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013268+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.013268+0000","last_peered":"2024-09-16T13:30:53.013268+0000","last_clean":"2024-09-16T13:30:53.013268+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.013268+0000","last_undegraded":"2024-09-16T13:30:53.013268+0000","last_fullsized":"2024-09-16T13:30:53.013268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2516,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195638+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:52.195638+0000","last_peered":"2024-09-16T13:30:52.195638+0000","last_clean":"2024-09-16T13:30:52.195638+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:52.195638+0000","last_undegraded":"2024-09-16T13:30:52.195638+0000","last_fullsized":"2024-09-16T13:30:52.195638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4818,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374909+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:52.374909+0000","last_peered":"2024-09-16T13:30:52.374909+0000","last_clean":"2024-09-16T13:30:52.374909+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:52.374909+0000","last_undegraded":"2024-09-16T13:30:52.374909+0000","last_fullsized":"2024-09-16T13:30:52.374909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4597,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195569+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:52.195569+0000","last_peered":"2024-09-16T13:30:52.195569+0000","last_clean":"2024-09-16T13:30:52.195569+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:52.195569+0000","last_undegraded":"2024-09-16T13:30:52.195569+0000","last_fullsized":"2024-09-16T13:30:52.195569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5510,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195686+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:52.195686+0000","last_peered":"2024-09-16T13:30:52.195686+0000","last_clean":"2024-09-16T13:30:52.195686+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:52.195686+0000","last_undegraded":"2024-09-16T13:30:52.195686+0000","last_fullsized":"2024-09-16T13:30:52.195686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5587,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375030+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:52.375030+0000","last_peered":"2024-09-16T13:30:52.375030+0000","last_clean":"2024-09-16T13:30:52.375030+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:52.375030+0000","last_undegraded":"2024-09-16T13:30:52.375030+0000","last_fullsized":"2024-09-16T13:30:52.375030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1804,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375050+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:52.375050+0000","last_peered":"2024-09-16T13:30:52.375050+0000","last_clean":"2024-09-16T13:30:52.375050+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:52.375050+0000","last_undegraded":"2024-09-16T13:30:52.375050+0000","last_fullsized":"2024-09-16T13:30:52.375050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671727+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:53.671727+0000","last_peered":"2024-09-16T13:30:53.671727+0000","last_clean":"2024-09-16T13:30:53.671727+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:53.671727+0000","last_undegraded":"2024-09-16T13:30:53.671727+0000","last_fullsized":"2024-09-16T13:30:53.671727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303492,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068348,"statfs":{"total":96636764160,"available":96325988352,"internally_reserved":0,"allocated":2019328,"data_stored":1600134,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.497}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.497}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.378},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1009951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.439},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.501},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52600000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509479,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.246},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.46999999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299292,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072548,"statfs":{"total":96636764160,"available":96330289152,"internally_reserved":0,"allocated":1978368,"data_stored":1009766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.502}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.376}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.48399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53000000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1982464,"data_stored":1009785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.378}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.56000000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299352,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072488,"statfs":{"total":96636764160,"available":96330227712,"internally_reserved":0,"allocated":2039808,"data_stored":1600217,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.438}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:58.880 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:30:58.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:30:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:59.143 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:30:59.154 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-16T13:30:57.869183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18527,"ondisk_log_size":18527,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436092,"kb_used_data":15620,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538628,"statfs":{"total":773094113280,"available":770599555072,"internally_reserved":0,"allocated":15994880,"data_stored":9272199,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000939"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2264,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195747+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:52.195747+0000","last_peered":"2024-09-16T13:30:52.195747+0000","last_clean":"2024-09-16T13:30:52.195747+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:52.195747+0000","last_undegraded":"2024-09-16T13:30:52.195747+0000","last_fullsized":"2024-09-16T13:30:52.195747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11845,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012799+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:53.012799+0000","last_peered":"2024-09-16T13:30:53.012799+0000","last_clean":"2024-09-16T13:30:53.012799+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:53.012799+0000","last_undegraded":"2024-09-16T13:30:53.012799+0000","last_fullsized":"2024-09-16T13:30:53.012799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.050755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1680,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:52.197025+0000","last_peered":"2024-09-16T13:30:52.197025+0000","last_clean":"2024-09-16T13:30:52.197025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:52.197025+0000","last_undegraded":"2024-09-16T13:30:52.197025+0000","last_fullsized":"2024-09-16T13:30:52.197025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4601,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671843+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:53.671843+0000","last_peered":"2024-09-16T13:30:53.671843+0000","last_clean":"2024-09-16T13:30:53.671843+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:53.671843+0000","last_undegraded":"2024-09-16T13:30:53.671843+0000","last_fullsized":"2024-09-16T13:30:53.671843+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:41.083009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2377,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374989+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:52.374989+0000","last_peered":"2024-09-16T13:30:52.374989+0000","last_clean":"2024-09-16T13:30:52.374989+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:52.374989+0000","last_undegraded":"2024-09-16T13:30:52.374989+0000","last_fullsized":"2024-09-16T13:30:52.374989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4334,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374920+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:52.374920+0000","last_peered":"2024-09-16T13:30:52.374920+0000","last_clean":"2024-09-16T13:30:52.374920+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:52.374920+0000","last_undegraded":"2024-09-16T13:30:52.374920+0000","last_fullsized":"2024-09-16T13:30:52.374920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3515,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197083+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:52.197083+0000","last_peered":"2024-09-16T13:30:52.197083+0000","last_clean":"2024-09-16T13:30:52.197083+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:52.197083+0000","last_undegraded":"2024-09-16T13:30:52.197083+0000","last_fullsized":"2024-09-16T13:30:52.197083+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3577,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374915+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:52.374915+0000","last_peered":"2024-09-16T13:30:52.374915+0000","last_clean":"2024-09-16T13:30:52.374915+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:52.374915+0000","last_undegraded":"2024-09-16T13:30:52.374915+0000","last_fullsized":"2024-09-16T13:30:52.374915+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3267,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.197135+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:52.197135+0000","last_peered":"2024-09-16T13:30:52.197135+0000","last_clean":"2024-09-16T13:30:52.197135+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:52.197135+0000","last_undegraded":"2024-09-16T13:30:52.197135+0000","last_fullsized":"2024-09-16T13:30:52.197135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2883,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012849+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:53.012849+0000","last_peered":"2024-09-16T13:30:53.012849+0000","last_clean":"2024-09-16T13:30:53.012849+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:53.012849+0000","last_undegraded":"2024-09-16T13:30:53.012849+0000","last_fullsized":"2024-09-16T13:30:53.012849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:17.041790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3513,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196998+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:52.196998+0000","last_peered":"2024-09-16T13:30:52.196998+0000","last_clean":"2024-09-16T13:30:52.196998+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:52.196998+0000","last_undegraded":"2024-09-16T13:30:52.196998+0000","last_fullsized":"2024-09-16T13:30:52.196998+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2023,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375068+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:52.375068+0000","last_peered":"2024-09-16T13:30:52.375068+0000","last_clean":"2024-09-16T13:30:52.375068+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:52.375068+0000","last_undegraded":"2024-09-16T13:30:52.375068+0000","last_fullsized":"2024-09-16T13:30:52.375068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4363,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201629+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:53.201629+0000","last_peered":"2024-09-16T13:30:53.201629+0000","last_clean":"2024-09-16T13:30:53.201629+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:53.201629+0000","last_undegraded":"2024-09-16T13:30:53.201629+0000","last_fullsized":"2024-09-16T13:30:53.201629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:15.353342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4867,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196996+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:52.196996+0000","last_peered":"2024-09-16T13:30:52.196996+0000","last_clean":"2024-09-16T13:30:52.196996+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:52.196996+0000","last_undegraded":"2024-09-16T13:30:52.196996+0000","last_fullsized":"2024-09-16T13:30:52.196996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3938,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196714+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:52.196714+0000","last_peered":"2024-09-16T13:30:52.196714+0000","last_clean":"2024-09-16T13:30:52.196714+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:52.196714+0000","last_undegraded":"2024-09-16T13:30:52.196714+0000","last_fullsized":"2024-09-16T13:30:52.196714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195771+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:52.195771+0000","last_peered":"2024-09-16T13:30:52.195771+0000","last_clean":"2024-09-16T13:30:52.195771+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:52.195771+0000","last_undegraded":"2024-09-16T13:30:52.195771+0000","last_fullsized":"2024-09-16T13:30:52.195771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3621,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374947+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:52.374947+0000","last_peered":"2024-09-16T13:30:52.374947+0000","last_clean":"2024-09-16T13:30:52.374947+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:52.374947+0000","last_undegraded":"2024-09-16T13:30:52.374947+0000","last_fullsized":"2024-09-16T13:30:52.374947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201743+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201743+0000","last_peered":"2024-09-16T13:30:53.201743+0000","last_clean":"2024-09-16T13:30:53.201743+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:30:53.201743+0000","last_undegraded":"2024-09-16T13:30:53.201743+0000","last_fullsized":"2024-09-16T13:30:53.201743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:41:52.148063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2654,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.196913+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:52.196913+0000","last_peered":"2024-09-16T13:30:52.196913+0000","last_clean":"2024-09-16T13:30:52.196913+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:52.196913+0000","last_undegraded":"2024-09-16T13:30:52.196913+0000","last_fullsized":"2024-09-16T13:30:52.196913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195591+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:52.195591+0000","last_peered":"2024-09-16T13:30:52.195591+0000","last_clean":"2024-09-16T13:30:52.195591+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:52.195591+0000","last_undegraded":"2024-09-16T13:30:52.195591+0000","last_fullsized":"2024-09-16T13:30:52.195591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4247,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012756+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:53.012756+0000","last_peered":"2024-09-16T13:30:53.012756+0000","last_clean":"2024-09-16T13:30:53.012756+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:53.012756+0000","last_undegraded":"2024-09-16T13:30:53.012756+0000","last_fullsized":"2024-09-16T13:30:53.012756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:50.144533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201795+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:53.201795+0000","last_peered":"2024-09-16T13:30:53.201795+0000","last_clean":"2024-09-16T13:30:53.201795+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:53.201795+0000","last_undegraded":"2024-09-16T13:30:53.201795+0000","last_fullsized":"2024-09-16T13:30:53.201795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:17.607911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3497,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201742+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:53.201742+0000","last_peered":"2024-09-16T13:30:53.201742+0000","last_clean":"2024-09-16T13:30:53.201742+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:53.201742+0000","last_undegraded":"2024-09-16T13:30:53.201742+0000","last_fullsized":"2024-09-16T13:30:53.201742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:13.434914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":145,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375070+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:52.375070+0000","last_peered":"2024-09-16T13:30:52.375070+0000","last_clean":"2024-09-16T13:30:52.375070+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:52.375070+0000","last_undegraded":"2024-09-16T13:30:52.375070+0000","last_fullsized":"2024-09-16T13:30:52.375070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4566,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201694+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:53.201694+0000","last_peered":"2024-09-16T13:30:53.201694+0000","last_clean":"2024-09-16T13:30:53.201694+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:53.201694+0000","last_undegraded":"2024-09-16T13:30:53.201694+0000","last_fullsized":"2024-09-16T13:30:53.201694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:31.982715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201531+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:53.201531+0000","last_peered":"2024-09-16T13:30:53.201531+0000","last_clean":"2024-09-16T13:30:53.201531+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:53.201531+0000","last_undegraded":"2024-09-16T13:30:53.201531+0000","last_fullsized":"2024-09-16T13:30:53.201531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671758+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:53.671758+0000","last_peered":"2024-09-16T13:30:53.671758+0000","last_clean":"2024-09-16T13:30:53.671758+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:53.671758+0000","last_undegraded":"2024-09-16T13:30:53.671758+0000","last_fullsized":"2024-09-16T13:30:53.671758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:08:03.534959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3470,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201711+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:53.201711+0000","last_peered":"2024-09-16T13:30:53.201711+0000","last_clean":"2024-09-16T13:30:53.201711+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:53.201711+0000","last_undegraded":"2024-09-16T13:30:53.201711+0000","last_fullsized":"2024-09-16T13:30:53.201711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:15.191850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'3","reported_seq":150,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202251+0000","last_change":"2024-09-16T13:28:17.927609+0000","last_active":"2024-09-16T13:30:53.202251+0000","last_peered":"2024-09-16T13:30:53.202251+0000","last_clean":"2024-09-16T13:30:53.202251+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:30:53.202251+0000","last_undegraded":"2024-09-16T13:30:53.202251+0000","last_fullsized":"2024-09-16T13:30:53.202251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:03.040570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013330+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.013330+0000","last_peered":"2024-09-16T13:30:53.013330+0000","last_clean":"2024-09-16T13:30:53.013330+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.013330+0000","last_undegraded":"2024-09-16T13:30:53.013330+0000","last_fullsized":"2024-09-16T13:30:53.013330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4081,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.012913+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:53.012913+0000","last_peered":"2024-09-16T13:30:53.012913+0000","last_clean":"2024-09-16T13:30:53.012913+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:53.012913+0000","last_undegraded":"2024-09-16T13:30:53.012913+0000","last_fullsized":"2024-09-16T13:30:53.012913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:20:48.106629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":151,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671937+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.671937+0000","last_peered":"2024-09-16T13:30:53.671937+0000","last_clean":"2024-09-16T13:30:53.671937+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.671937+0000","last_undegraded":"2024-09-16T13:30:53.671937+0000","last_fullsized":"2024-09-16T13:30:53.671937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":110,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.013268+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.013268+0000","last_peered":"2024-09-16T13:30:53.013268+0000","last_clean":"2024-09-16T13:30:53.013268+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.013268+0000","last_undegraded":"2024-09-16T13:30:53.013268+0000","last_fullsized":"2024-09-16T13:30:53.013268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2516,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195638+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:52.195638+0000","last_peered":"2024-09-16T13:30:52.195638+0000","last_clean":"2024-09-16T13:30:52.195638+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:52.195638+0000","last_undegraded":"2024-09-16T13:30:52.195638+0000","last_fullsized":"2024-09-16T13:30:52.195638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4818,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.374909+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:52.374909+0000","last_peered":"2024-09-16T13:30:52.374909+0000","last_clean":"2024-09-16T13:30:52.374909+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:52.374909+0000","last_undegraded":"2024-09-16T13:30:52.374909+0000","last_fullsized":"2024-09-16T13:30:52.374909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4597,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195569+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:52.195569+0000","last_peered":"2024-09-16T13:30:52.195569+0000","last_clean":"2024-09-16T13:30:52.195569+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:52.195569+0000","last_undegraded":"2024-09-16T13:30:52.195569+0000","last_fullsized":"2024-09-16T13:30:52.195569+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5510,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.195686+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:52.195686+0000","last_peered":"2024-09-16T13:30:52.195686+0000","last_clean":"2024-09-16T13:30:52.195686+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:52.195686+0000","last_undegraded":"2024-09-16T13:30:52.195686+0000","last_fullsized":"2024-09-16T13:30:52.195686+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5587,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375030+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:52.375030+0000","last_peered":"2024-09-16T13:30:52.375030+0000","last_clean":"2024-09-16T13:30:52.375030+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:52.375030+0000","last_undegraded":"2024-09-16T13:30:52.375030+0000","last_fullsized":"2024-09-16T13:30:52.375030+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1804,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-16T13:30:52.375050+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:52.375050+0000","last_peered":"2024-09-16T13:30:52.375050+0000","last_clean":"2024-09-16T13:30:52.375050+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:52.375050+0000","last_undegraded":"2024-09-16T13:30:52.375050+0000","last_fullsized":"2024-09-16T13:30:52.375050+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3444,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671727+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:53.671727+0000","last_peered":"2024-09-16T13:30:53.671727+0000","last_clean":"2024-09-16T13:30:53.671727+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:53.671727+0000","last_undegraded":"2024-09-16T13:30:53.671727+0000","last_fullsized":"2024-09-16T13:30:53.671727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:49.582974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303492,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068348,"statfs":{"total":96636764160,"available":96325988352,"internally_reserved":0,"allocated":2019328,"data_stored":1600134,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.497}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509481,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.622}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.497}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.378},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509480,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1009951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.439},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.501},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52600000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509479,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.246},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.46999999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299292,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072548,"statfs":{"total":96636764160,"available":96330289152,"internally_reserved":0,"allocated":1978368,"data_stored":1009766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.61599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.502}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56399999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.376}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.48399999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.53000000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1982464,"data_stored":1009785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.378}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.56000000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299352,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072488,"statfs":{"total":96636764160,"available":96330227712,"internally_reserved":0,"allocated":2039808,"data_stored":1600217,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.438}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:30:59.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:30:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:30:59.416 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":166,"fsid":"0cf46de4-f58b-4e68-a8b9-534bb33f8d72","created":"2024-09-16T13:27:50.121006+0000","modified":"2024-09-16T13:30:54.203791+0000","last_up_change":"2024-09-16T13:28:03.264780+0000","last_in_change":"2024-09-16T13:29:06.165279+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:28:03.867975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:28:08.815950+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-09-16T13:28:16.900065+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"27","last_force_op_resend_prenautilus":"27","last_force_op_resend_preluminous":"27","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":25,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"pool_client.0","create_time":"2024-09-16T13:28:25.583589+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"daa6f01d-c106-4b1d-8063-1852db16dec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":33372},{"type":"v1","addr":"172.21.15.76:6817","nonce":33372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":33372},{"type":"v1","addr":"172.21.15.76:6819","nonce":33372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":33372},{"type":"v1","addr":"172.21.15.76:6823","nonce":33372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":33372},{"type":"v1","addr":"172.21.15.76:6821","nonce":33372}]},"public_addr":"172.21.15.76:6817/33372","cluster_addr":"172.21.15.76:6819/33372","heartbeat_back_addr":"172.21.15.76:6823/33372","heartbeat_front_addr":"172.21.15.76:6821/33372","state":["exists","up"]},{"osd":1,"uuid":"70bfb231-8371-4267-83d7-8c7bdf4d122b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":33371},{"type":"v1","addr":"172.21.15.76:6801","nonce":33371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":33371},{"type":"v1","addr":"172.21.15.76:6803","nonce":33371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":33371},{"type":"v1","addr":"172.21.15.76:6807","nonce":33371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":33371},{"type":"v1","addr":"172.21.15.76:6805","nonce":33371}]},"public_addr":"172.21.15.76:6801/33371","cluster_addr":"172.21.15.76:6803/33371","heartbeat_back_addr":"172.21.15.76:6807/33371","heartbeat_front_addr":"172.21.15.76:6805/33371","state":["exists","up"]},{"osd":2,"uuid":"a4447a8c-bda5-4387-bc24-77c6cc28374b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":33387},{"type":"v1","addr":"172.21.15.76:6809","nonce":33387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":33387},{"type":"v1","addr":"172.21.15.76:6811","nonce":33387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":33387},{"type":"v1","addr":"172.21.15.76:6815","nonce":33387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":33387},{"type":"v1","addr":"172.21.15.76:6813","nonce":33387}]},"public_addr":"172.21.15.76:6809/33387","cluster_addr":"172.21.15.76:6811/33387","heartbeat_back_addr":"172.21.15.76:6815/33387","heartbeat_front_addr":"172.21.15.76:6813/33387","state":["exists","up"]},{"osd":3,"uuid":"50031c12-2464-4373-b82a-2d4da1dae766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":33388},{"type":"v1","addr":"172.21.15.76:6825","nonce":33388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":33388},{"type":"v1","addr":"172.21.15.76:6827","nonce":33388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":33388},{"type":"v1","addr":"172.21.15.76:6831","nonce":33388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":33388},{"type":"v1","addr":"172.21.15.76:6829","nonce":33388}]},"public_addr":"172.21.15.76:6825/33388","cluster_addr":"172.21.15.76:6827/33388","heartbeat_back_addr":"172.21.15.76:6831/33388","heartbeat_front_addr":"172.21.15.76:6829/33388","state":["exists","up"]},{"osd":4,"uuid":"24b2d993-ea4d-406e-adbd-e06d278de26d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":33329},{"type":"v1","addr":"172.21.15.163:6809","nonce":33329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":33329},{"type":"v1","addr":"172.21.15.163:6813","nonce":33329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6821","nonce":33329},{"type":"v1","addr":"172.21.15.163:6824","nonce":33329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":33329},{"type":"v1","addr":"172.21.15.163:6818","nonce":33329}]},"public_addr":"172.21.15.163:6809/33329","cluster_addr":"172.21.15.163:6813/33329","heartbeat_back_addr":"172.21.15.163:6824/33329","heartbeat_front_addr":"172.21.15.163:6818/33329","state":["exists","up"]},{"osd":5,"uuid":"d68602fa-7b2b-43fc-96ba-f807ca71872e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6817","nonce":33357},{"type":"v1","addr":"172.21.15.163:6820","nonce":33357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6823","nonce":33357},{"type":"v1","addr":"172.21.15.163:6827","nonce":33357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":33357},{"type":"v1","addr":"172.21.15.163:6831","nonce":33357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":33357},{"type":"v1","addr":"172.21.15.163:6829","nonce":33357}]},"public_addr":"172.21.15.163:6820/33357","cluster_addr":"172.21.15.163:6827/33357","heartbeat_back_addr":"172.21.15.163:6831/33357","heartbeat_front_addr":"172.21.15.163:6829/33357","state":["exists","up"]},{"osd":6,"uuid":"5f02e46d-718f-454b-99c5-245ded452730","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":33330},{"type":"v1","addr":"172.21.15.163:6801","nonce":33330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":33330},{"type":"v1","addr":"172.21.15.163:6803","nonce":33330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":33330},{"type":"v1","addr":"172.21.15.163:6807","nonce":33330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":33330},{"type":"v1","addr":"172.21.15.163:6805","nonce":33330}]},"public_addr":"172.21.15.163:6801/33330","cluster_addr":"172.21.15.163:6803/33330","heartbeat_back_addr":"172.21.15.163:6807/33330","heartbeat_front_addr":"172.21.15.163:6805/33330","state":["exists","up"]},{"osd":7,"uuid":"ea09d724-4028-4cd4-b5b5-9d0122dd5495","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":33331},{"type":"v1","addr":"172.21.15.163:6812","nonce":33331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":33331},{"type":"v1","addr":"172.21.15.163:6816","nonce":33331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6825","nonce":33331},{"type":"v1","addr":"172.21.15.163:6826","nonce":33331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6819","nonce":33331},{"type":"v1","addr":"172.21.15.163:6822","nonce":33331}]},"public_addr":"172.21.15.163:6812/33331","cluster_addr":"172.21.15.163:6816/33331","heartbeat_back_addr":"172.21.15.163:6826/33331","heartbeat_front_addr":"172.21.15.163:6822/33331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:28:01.464981+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.067355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.138713+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.125791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.187216+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.205813+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.086803+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:27:58.207997+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[6,1]},{"pgid":"4.b","osds":[6,4]},{"pgid":"4.d","osds":[5,0]}],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:31:00.430 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T13:31:00.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:00.554 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:00.555 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:00.555 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:00.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T13:31:00.826 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:31:00.839 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T13:31:00.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:00.955 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:00.956 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:00.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T13:31:01.225 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:31:01.239 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T13:31:01.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:01.355 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:01.356 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:01.356 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:01.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T13:31:01.625 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:31:01.637 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T13:31:01.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:01.754 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:01.754 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:01.755 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:01.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T13:31:02.025 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T13:31:02.038 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T13:31:02.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:02.156 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:02.157 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:02.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T13:31:02.427 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T13:31:02.440 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T13:31:02.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:02.557 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:02.557 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:02.557 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:02.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T13:31:02.826 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T13:31:02.839 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T13:31:02.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:02.957 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:02.957 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:02.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T13:31:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T13:31:03.240 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T13:31:03.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:31:03.357 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T13:31:03.357 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:31:03.357 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T13:31:03.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T13:31:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T13:31:03.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:31:03.907 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:31:03.907 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:31:03.918 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T13:31:03.870299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18530,"ondisk_log_size":18530,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436148,"kb_used_data":15696,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538572,"statfs":{"total":773094113280,"available":770599497728,"internally_reserved":0,"allocated":16072704,"data_stored":9310292,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001800"},"pg_stats":[{"pgid":"4.1f","version":"145'367","reported_seq":2265,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672450+0000","last_change":"2024-09-16T13:30:32.932878+0000","last_active":"2024-09-16T13:30:53.672450+0000","last_peered":"2024-09-16T13:30:53.672450+0000","last_clean":"2024-09-16T13:30:53.672450+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:30:53.672450+0000","last_undegraded":"2024-09-16T13:30:53.672450+0000","last_fullsized":"2024-09-16T13:30:53.672450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:31:15.101059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"143'1095","reported_seq":11847,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972318+0000","last_change":"2024-09-16T13:30:32.934449+0000","last_active":"2024-09-16T13:30:55.972318+0000","last_peered":"2024-09-16T13:30:55.972318+0000","last_clean":"2024-09-16T13:30:55.972318+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:30:55.972318+0000","last_undegraded":"2024-09-16T13:30:55.972318+0000","last_fullsized":"2024-09-16T13:30:55.972318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1681,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672025+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:53.672025+0000","last_peered":"2024-09-16T13:30:53.672025+0000","last_clean":"2024-09-16T13:30:53.672025+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:53.672025+0000","last_undegraded":"2024-09-16T13:30:53.672025+0000","last_fullsized":"2024-09-16T13:30:53.672025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:02.821831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"145'510","reported_seq":4602,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208840+0000","last_change":"2024-09-16T13:30:32.930625+0000","last_active":"2024-09-16T13:30:54.208840+0000","last_peered":"2024-09-16T13:30:54.208840+0000","last_clean":"2024-09-16T13:30:54.208840+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:30:54.208840+0000","last_undegraded":"2024-09-16T13:30:54.208840+0000","last_fullsized":"2024-09-16T13:30:54.208840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2378,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201943+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:53.201943+0000","last_peered":"2024-09-16T13:30:53.201943+0000","last_clean":"2024-09-16T13:30:53.201943+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:53.201943+0000","last_undegraded":"2024-09-16T13:30:53.201943+0000","last_fullsized":"2024-09-16T13:30:53.201943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:17.806149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4335,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201924+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:53.201924+0000","last_peered":"2024-09-16T13:30:53.201924+0000","last_clean":"2024-09-16T13:30:53.201924+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:53.201924+0000","last_undegraded":"2024-09-16T13:30:53.201924+0000","last_fullsized":"2024-09-16T13:30:53.201924+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:34.723070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3516,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671888+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:53.671888+0000","last_peered":"2024-09-16T13:30:53.671888+0000","last_clean":"2024-09-16T13:30:53.671888+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:53.671888+0000","last_undegraded":"2024-09-16T13:30:53.671888+0000","last_fullsized":"2024-09-16T13:30:53.671888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:49.381329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"145'416","reported_seq":3578,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201911+0000","last_change":"2024-09-16T13:30:32.937103+0000","last_active":"2024-09-16T13:30:53.201911+0000","last_peered":"2024-09-16T13:30:53.201911+0000","last_clean":"2024-09-16T13:30:53.201911+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:30:53.201911+0000","last_undegraded":"2024-09-16T13:30:53.201911+0000","last_fullsized":"2024-09-16T13:30:53.201911+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:59.385558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3268,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672060+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:53.672060+0000","last_peered":"2024-09-16T13:30:53.672060+0000","last_clean":"2024-09-16T13:30:53.672060+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:53.672060+0000","last_undegraded":"2024-09-16T13:30:53.672060+0000","last_fullsized":"2024-09-16T13:30:53.672060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:12.620705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"143'654","reported_seq":2885,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972387+0000","last_change":"2024-09-16T13:30:32.934763+0000","last_active":"2024-09-16T13:30:55.972387+0000","last_peered":"2024-09-16T13:30:55.972387+0000","last_clean":"2024-09-16T13:30:55.972387+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:30:55.972387+0000","last_undegraded":"2024-09-16T13:30:55.972387+0000","last_fullsized":"2024-09-16T13:30:55.972387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4137,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294246+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:53.294246+0000","last_peered":"2024-09-16T13:30:53.294246+0000","last_clean":"2024-09-16T13:30:53.294246+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:53.294246+0000","last_undegraded":"2024-09-16T13:30:53.294246+0000","last_fullsized":"2024-09-16T13:30:53.294246+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:47.663271+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"143'489","reported_seq":3514,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671984+0000","last_change":"2024-09-16T13:30:32.940780+0000","last_active":"2024-09-16T13:30:53.671984+0000","last_peered":"2024-09-16T13:30:53.671984+0000","last_clean":"2024-09-16T13:30:53.671984+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:30:53.671984+0000","last_undegraded":"2024-09-16T13:30:53.671984+0000","last_fullsized":"2024-09-16T13:30:53.671984+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:11.810479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"145'377","reported_seq":2024,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202047+0000","last_change":"2024-09-16T13:30:32.937505+0000","last_active":"2024-09-16T13:30:53.202047+0000","last_peered":"2024-09-16T13:30:53.202047+0000","last_clean":"2024-09-16T13:30:53.202047+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:30:53.202047+0000","last_undegraded":"2024-09-16T13:30:53.202047+0000","last_fullsized":"2024-09-16T13:30:53.202047+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:08.972860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"144'827","reported_seq":4364,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.209002+0000","last_change":"2024-09-16T13:30:32.931027+0000","last_active":"2024-09-16T13:30:54.209002+0000","last_peered":"2024-09-16T13:30:54.209002+0000","last_clean":"2024-09-16T13:30:54.209002+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:30:54.209002+0000","last_undegraded":"2024-09-16T13:30:54.209002+0000","last_fullsized":"2024-09-16T13:30:54.209002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"147'785","reported_seq":4868,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671875+0000","last_change":"2024-09-16T13:30:32.941446+0000","last_active":"2024-09-16T13:30:53.671875+0000","last_peered":"2024-09-16T13:30:53.671875+0000","last_clean":"2024-09-16T13:30:53.671875+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:30:53.671875+0000","last_undegraded":"2024-09-16T13:30:53.671875+0000","last_fullsized":"2024-09-16T13:30:53.671875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:26.918347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3939,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671771+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:53.671771+0000","last_peered":"2024-09-16T13:30:53.671771+0000","last_clean":"2024-09-16T13:30:53.671771+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:53.671771+0000","last_undegraded":"2024-09-16T13:30:53.671771+0000","last_fullsized":"2024-09-16T13:30:53.671771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:43.552482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":174,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672448+0000","last_change":"2024-09-16T13:29:07.182225+0000","last_active":"2024-09-16T13:30:53.672448+0000","last_peered":"2024-09-16T13:30:53.672448+0000","last_clean":"2024-09-16T13:30:53.672448+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:30:53.672448+0000","last_undegraded":"2024-09-16T13:30:53.672448+0000","last_fullsized":"2024-09-16T13:30:53.672448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:51.930321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3622,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201922+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:53.201922+0000","last_peered":"2024-09-16T13:30:53.201922+0000","last_clean":"2024-09-16T13:30:53.201922+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:53.201922+0000","last_undegraded":"2024-09-16T13:30:53.201922+0000","last_fullsized":"2024-09-16T13:30:53.201922+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:26:01.542711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.198348+0000","last_change":"2024-09-16T13:31:02.198329+0000","last_active":"2024-09-16T13:31:02.198348+0000","last_peered":"2024-09-16T13:31:02.198348+0000","last_clean":"2024-09-16T13:31:02.198348+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:31:02.198348+0000","last_undegraded":"2024-09-16T13:31:02.198348+0000","last_fullsized":"2024-09-16T13:31:02.198348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.198261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:06.779026+0000","scrub_duration":0.0015109539999999999,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"144'360","reported_seq":2655,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.671798+0000","last_change":"2024-09-16T13:30:32.941602+0000","last_active":"2024-09-16T13:30:53.671798+0000","last_peered":"2024-09-16T13:30:53.671798+0000","last_clean":"2024-09-16T13:30:53.671798+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:30:53.671798+0000","last_undegraded":"2024-09-16T13:30:53.671798+0000","last_fullsized":"2024-09-16T13:30:53.671798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:10.460541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672263+0000","last_change":"2024-09-16T13:28:28.093355+0000","last_active":"2024-09-16T13:30:53.672263+0000","last_peered":"2024-09-16T13:30:53.672263+0000","last_clean":"2024-09-16T13:30:53.672263+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:30:53.672263+0000","last_undegraded":"2024-09-16T13:30:53.672263+0000","last_fullsized":"2024-09-16T13:30:53.672263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:47.227839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"147'822","reported_seq":4249,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972525+0000","last_change":"2024-09-16T13:30:32.935230+0000","last_active":"2024-09-16T13:30:55.972525+0000","last_peered":"2024-09-16T13:30:55.972525+0000","last_clean":"2024-09-16T13:30:55.972525+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:30:55.972525+0000","last_undegraded":"2024-09-16T13:30:55.972525+0000","last_fullsized":"2024-09-16T13:30:55.972525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.209044+0000","last_change":"2024-09-16T13:28:12.312172+0000","last_active":"2024-09-16T13:30:54.209044+0000","last_peered":"2024-09-16T13:30:54.209044+0000","last_clean":"2024-09-16T13:30:54.209044+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:30:54.209044+0000","last_undegraded":"2024-09-16T13:30:54.209044+0000","last_fullsized":"2024-09-16T13:30:54.209044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"144'593","reported_seq":3498,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.209039+0000","last_change":"2024-09-16T13:30:32.931498+0000","last_active":"2024-09-16T13:30:54.209039+0000","last_peered":"2024-09-16T13:30:54.209039+0000","last_clean":"2024-09-16T13:30:54.209039+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:30:54.209039+0000","last_undegraded":"2024-09-16T13:30:54.209039+0000","last_fullsized":"2024-09-16T13:30:54.209039+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":146,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202117+0000","last_change":"2024-09-16T13:28:17.926596+0000","last_active":"2024-09-16T13:30:53.202117+0000","last_peered":"2024-09-16T13:30:53.202117+0000","last_clean":"2024-09-16T13:30:53.202117+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:30:53.202117+0000","last_undegraded":"2024-09-16T13:30:53.202117+0000","last_fullsized":"2024-09-16T13:30:53.202117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:13.688978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"22'2","reported_seq":113,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294210+0000","last_change":"2024-09-16T13:30:51.196388+0000","last_active":"2024-09-16T13:30:53.294210+0000","last_peered":"2024-09-16T13:30:53.294210+0000","last_clean":"2024-09-16T13:30:53.294210+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:30:53.294210+0000","last_undegraded":"2024-09-16T13:30:53.294210+0000","last_fullsized":"2024-09-16T13:30:53.294210+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:20.896465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"144'369","reported_seq":4567,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208993+0000","last_change":"2024-09-16T13:30:32.931055+0000","last_active":"2024-09-16T13:30:54.208993+0000","last_peered":"2024-09-16T13:30:54.208993+0000","last_clean":"2024-09-16T13:30:54.208993+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:30:54.208993+0000","last_undegraded":"2024-09-16T13:30:54.208993+0000","last_fullsized":"2024-09-16T13:30:54.208993+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208745+0000","last_change":"2024-09-16T13:28:17.924499+0000","last_active":"2024-09-16T13:30:54.208745+0000","last_peered":"2024-09-16T13:30:54.208745+0000","last_clean":"2024-09-16T13:30:54.208745+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:30:54.208745+0000","last_undegraded":"2024-09-16T13:30:54.208745+0000","last_fullsized":"2024-09-16T13:30:54.208745+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":158,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208833+0000","last_change":"2024-09-16T13:28:12.313196+0000","last_active":"2024-09-16T13:30:54.208833+0000","last_peered":"2024-09-16T13:30:54.208833+0000","last_clean":"2024-09-16T13:30:54.208833+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:30:54.208833+0000","last_undegraded":"2024-09-16T13:30:54.208833+0000","last_fullsized":"2024-09-16T13:30:54.208833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"143'485","reported_seq":3471,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208853+0000","last_change":"2024-09-16T13:30:32.930321+0000","last_active":"2024-09-16T13:30:54.208853+0000","last_peered":"2024-09-16T13:30:54.208853+0000","last_clean":"2024-09-16T13:30:54.208853+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:30:54.208853+0000","last_undegraded":"2024-09-16T13:30:54.208853+0000","last_fullsized":"2024-09-16T13:30:54.208853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"166'6","reported_seq":165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.992554+0000","last_change":"2024-09-16T13:31:02.992512+0000","last_active":"2024-09-16T13:31:02.992554+0000","last_peered":"2024-09-16T13:31:02.992554+0000","last_clean":"2024-09-16T13:31:02.992554+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:31:02.992554+0000","last_undegraded":"2024-09-16T13:31:02.992554+0000","last_fullsized":"2024-09-16T13:31:02.992554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6","last_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_deep_scrub":"166'6","last_deep_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.992436+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:49.262037+0000","scrub_duration":0.0040630290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":105,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672343+0000","last_change":"2024-09-16T13:29:07.184000+0000","last_active":"2024-09-16T13:30:53.672343+0000","last_peered":"2024-09-16T13:30:53.672343+0000","last_clean":"2024-09-16T13:30:53.672343+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:30:53.672343+0000","last_undegraded":"2024-09-16T13:30:53.672343+0000","last_fullsized":"2024-09-16T13:30:53.672343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:26.554828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"147'597","reported_seq":4083,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972438+0000","last_change":"2024-09-16T13:30:32.934821+0000","last_active":"2024-09-16T13:30:55.972438+0000","last_peered":"2024-09-16T13:30:55.972438+0000","last_clean":"2024-09-16T13:30:55.972438+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:30:55.972438+0000","last_undegraded":"2024-09-16T13:30:55.972438+0000","last_fullsized":"2024-09-16T13:30:55.972438+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"165'4","reported_seq":152,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.865248+0000","last_change":"2024-09-16T13:28:18.023463+0000","last_active":"2024-09-16T13:30:53.865248+0000","last_peered":"2024-09-16T13:30:53.865248+0000","last_clean":"2024-09-16T13:30:53.865248+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:30:53.865248+0000","last_undegraded":"2024-09-16T13:30:53.865248+0000","last_fullsized":"2024-09-16T13:30:53.865248+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:16.909933+0000","last_clean_scrub_stamp":"2024-09-16T13:28:16.909933+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:04.959439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":112,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294182+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:53.294182+0000","last_peered":"2024-09-16T13:30:53.294182+0000","last_clean":"2024-09-16T13:30:53.294182+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:53.294182+0000","last_undegraded":"2024-09-16T13:30:53.294182+0000","last_fullsized":"2024-09-16T13:30:53.294182+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:26:39.399541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672259+0000","last_change":"2024-09-16T13:29:07.189949+0000","last_active":"2024-09-16T13:30:53.672259+0000","last_peered":"2024-09-16T13:30:53.672259+0000","last_clean":"2024-09-16T13:30:53.672259+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:30:53.672259+0000","last_undegraded":"2024-09-16T13:30:53.672259+0000","last_fullsized":"2024-09-16T13:30:53.672259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:04.266697+0000","last_clean_scrub_stamp":"2024-09-16T13:28:04.266697+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:47.251475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"145'600","reported_seq":2517,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672363+0000","last_change":"2024-09-16T13:30:32.932936+0000","last_active":"2024-09-16T13:30:53.672363+0000","last_peered":"2024-09-16T13:30:53.672363+0000","last_clean":"2024-09-16T13:30:53.672363+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:30:53.672363+0000","last_undegraded":"2024-09-16T13:30:53.672363+0000","last_fullsized":"2024-09-16T13:30:53.672363+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:29:26.702318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"145'1259","reported_seq":5747,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294102+0000","last_change":"2024-09-16T13:30:51.198111+0000","last_active":"2024-09-16T13:30:53.294102+0000","last_peered":"2024-09-16T13:30:53.294102+0000","last_clean":"2024-09-16T13:30:53.294102+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:30:53.294102+0000","last_undegraded":"2024-09-16T13:30:53.294102+0000","last_fullsized":"2024-09-16T13:30:53.294102+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":1259,"log_dups_size":0,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:23.830101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4819,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.201912+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:53.201912+0000","last_peered":"2024-09-16T13:30:53.201912+0000","last_clean":"2024-09-16T13:30:53.201912+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:53.201912+0000","last_undegraded":"2024-09-16T13:30:53.201912+0000","last_fullsized":"2024-09-16T13:30:53.201912+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:09.551670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"144'572","reported_seq":4598,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672260+0000","last_change":"2024-09-16T13:30:32.933147+0000","last_active":"2024-09-16T13:30:53.672260+0000","last_peered":"2024-09-16T13:30:53.672260+0000","last_clean":"2024-09-16T13:30:53.672260+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:30:53.672260+0000","last_undegraded":"2024-09-16T13:30:53.672260+0000","last_fullsized":"2024-09-16T13:30:53.672260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:16:04.190270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"147'425","reported_seq":5511,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.672260+0000","last_change":"2024-09-16T13:30:37.099928+0000","last_active":"2024-09-16T13:30:53.672260+0000","last_peered":"2024-09-16T13:30:53.672260+0000","last_clean":"2024-09-16T13:30:53.672260+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:30:53.672260+0000","last_undegraded":"2024-09-16T13:30:53.672260+0000","last_fullsized":"2024-09-16T13:30:53.672260+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:37.551289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5588,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202039+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:53.202039+0000","last_peered":"2024-09-16T13:30:53.202039+0000","last_clean":"2024-09-16T13:30:53.202039+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:53.202039+0000","last_undegraded":"2024-09-16T13:30:53.202039+0000","last_fullsized":"2024-09-16T13:30:53.202039+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:18.690428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"145'457","reported_seq":3169,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.294266+0000","last_change":"2024-09-16T13:30:51.195272+0000","last_active":"2024-09-16T13:30:53.294266+0000","last_peered":"2024-09-16T13:30:53.294266+0000","last_clean":"2024-09-16T13:30:53.294266+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:30:53.294266+0000","last_undegraded":"2024-09-16T13:30:53.294266+0000","last_fullsized":"2024-09-16T13:30:53.294266+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:46.607217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"143'365","reported_seq":1805,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-16T13:30:53.202087+0000","last_change":"2024-09-16T13:30:32.936733+0000","last_active":"2024-09-16T13:30:53.202087+0000","last_peered":"2024-09-16T13:30:53.202087+0000","last_clean":"2024-09-16T13:30:53.202087+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:30:53.202087+0000","last_undegraded":"2024-09-16T13:30:53.202087+0000","last_fullsized":"2024-09-16T13:30:53.202087+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:23.297910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"144'848","reported_seq":3445,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.208649+0000","last_change":"2024-09-16T13:30:32.930026+0000","last_active":"2024-09-16T13:30:54.208649+0000","last_peered":"2024-09-16T13:30:54.208649+0000","last_clean":"2024-09-16T13:30:54.208649+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:30:54.208649+0000","last_undegraded":"2024-09-16T13:30:54.208649+0000","last_fullsized":"2024-09-16T13:30:54.208649+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18445,"ondisk_log_size":18445,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509480,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303500,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068340,"statfs":{"total":96636764160,"available":96325980160,"internally_reserved":0,"allocated":2031616,"data_stored":1606469,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.23400000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47599999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67600000000000005,"5min":0.879,"15min":0.879},"last":0.436},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60799999999999998,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0139999999999993,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.52300000000000002,"15min":0.53700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.621,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57399999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.48399999999999999,"15min":0.5},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57299999999999995,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.501},{"interface":"front","average":{"1min":0.434,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59999999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58699999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.47399999999999998,"15min":0.48599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.53900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61799999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59799999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.42399999999999999,"15min":0.434},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54700000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53000000000000003,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.7210000000000001,"5min":2.169,"15min":2.169},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56899999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46500000000000002,"15min":0.46999999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.627}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70099999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.629,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.495}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40200000000000002,"15min":0.41199999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.496,"5min":0.752,"15min":0.752},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55300000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.27400000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46899999999999997,"15min":0.47699999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.63600000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2002944,"data_stored":1016286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.46000000000000002,"15min":0.47999999999999998},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51500000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.496,"15min":0.51200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.42699999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.48499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64600000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40999999999999998,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.623}]},{"osd":7,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.73399999999999999,"5min":0.747,"15min":0.747},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69299999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.72299999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509480,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307632,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064208,"statfs":{"total":96636764160,"available":96321748992,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.28299999999999997,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.374,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.378},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.37,"15min":0.38100000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.54500000000000004,"5min":1.25,"15min":1.25},"last":1.645}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36799999999999999,"15min":0.38},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39000000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.37,"15min":0.377},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46300000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47499999999999998,"5min":1.093,"15min":1.093},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":1.675}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63800000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.629},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.40000000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":1.119,"15min":1.119},"last":0.68999999999999995}]}]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299300,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072540,"statfs":{"total":96636764160,"available":96330280960,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.72699999999999998,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.60699999999999998,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48599999999999999,"15min":0.49199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60299999999999998,"5min":1.286,"15min":1.286},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.58399999999999996,"15min":0.60499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.628,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.42599999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.45100000000000001,"15min":0.46700000000000003},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.53200000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.48699999999999999,"15min":0.50700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.504,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56699999999999995,"5min":1.008,"15min":1.008},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:30:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.437,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.50600000000000001,"15min":0.52100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58099999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.75900000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.57199999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.69599999999999995,"15min":0.75600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.56299999999999994,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.32300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.497,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.312},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.69199999999999995,"15min":0.752},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.621,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.70699999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.97599999999999998,"5min":1.194,"15min":1.194},"last":0.28199999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64800000000000002,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":1.23,"15min":1.23},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.85999999999999999,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.496}]},{"osd":7,"last update":"Mon Sep 16 13:30:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.876,"15min":0.93000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.61599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311784,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060056,"statfs":{"total":96636764160,"available":96317497344,"internally_reserved":0,"allocated":1994752,"data_stored":1016120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.42199999999999999,"15min":0.372},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.0999999999999996,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.245}]},{"osd":2,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.43099999999999999,"15min":0.38100000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2409999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38300000000000001,"15min":0.38600000000000001},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.47599999999999998,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.2720000000000002,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.81399999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.31,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.75,"15min":0.75},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.53200000000000003,"15min":0.54400000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.64000000000000001,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.3239999999999998,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.64800000000000002,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.64200000000000002,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:30:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.502,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.40699999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.65000000000000002,"5min":1.181,"15min":1.181},"last":0.54300000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299364,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072476,"statfs":{"total":96636764160,"available":96330215424,"internally_reserved":0,"allocated":2056192,"data_stored":1606635,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33400000000000002},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.45900000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.224},{"interface":"front","average":{"1min":0.315,"5min":0.35599999999999998,"15min":0.36299999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32600000000000001,"15min":0.32900000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42399999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.44,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.23999999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.40000000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.378},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.28000000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.55100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34499999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.53300000000000003,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57099999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63300000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.53400000000000003,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.55600000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.90400000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64200000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.622}]},{"osd":7,"last update":"Mon Sep 16 13:30:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.56999999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.925 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.926 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-16T13:28:16.909933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.927 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-16T13:28:16.909933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-16T13:28:16.909933+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.928 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T13:28:04.266697+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=4, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:03.929 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:31:23.931 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:31:24.212 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:31:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:31:24.224 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-16T13:31:23.875024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18737,"ondisk_log_size":18737,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2436112,"kb_used_data":15696,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":752538608,"statfs":{"total":773094113280,"available":770599534592,"internally_reserved":0,"allocated":16072704,"data_stored":9310292,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"4.1f","version":"166'378","reported_seq":2287,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.175418+0000","last_change":"2024-09-16T13:31:08.175392+0000","last_active":"2024-09-16T13:31:08.175418+0000","last_peered":"2024-09-16T13:31:08.175418+0000","last_clean":"2024-09-16T13:31:08.175418+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:31:08.175418+0000","last_undegraded":"2024-09-16T13:31:08.175418+0000","last_fullsized":"2024-09-16T13:31:08.175418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'378","last_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_deep_scrub":"166'378","last_deep_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.175307+0000","objects_scrubbed":11,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:49.299655+0000","scrub_duration":0.010270289,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"166'1105","reported_seq":11867,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.010295+0000","last_change":"2024-09-16T13:31:16.010249+0000","last_active":"2024-09-16T13:31:16.010295+0000","last_peered":"2024-09-16T13:31:16.010295+0000","last_clean":"2024-09-16T13:31:16.010295+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:31:16.010295+0000","last_undegraded":"2024-09-16T13:31:16.010295+0000","last_fullsized":"2024-09-16T13:31:16.010295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1105","last_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_deep_scrub":"166'1105","last_deep_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.010149+0000","objects_scrubbed":10,"log_size":1105,"log_dups_size":0,"ondisk_log_size":1105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:32.387613+0000","scrub_duration":0.009411081,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"144'448","reported_seq":1682,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972610+0000","last_change":"2024-09-16T13:30:32.941817+0000","last_active":"2024-09-16T13:30:55.972610+0000","last_peered":"2024-09-16T13:30:55.972610+0000","last_clean":"2024-09-16T13:30:55.972610+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:30:55.972610+0000","last_undegraded":"2024-09-16T13:30:55.972610+0000","last_fullsized":"2024-09-16T13:30:55.972610+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"166'515","reported_seq":4617,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.637148+0000","last_change":"2024-09-16T13:31:13.637117+0000","last_active":"2024-09-16T13:31:13.637148+0000","last_peered":"2024-09-16T13:31:13.637148+0000","last_clean":"2024-09-16T13:31:13.637148+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:31:13.637148+0000","last_undegraded":"2024-09-16T13:31:13.637148+0000","last_fullsized":"2024-09-16T13:31:13.637148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'515","last_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_deep_scrub":"166'515","last_deep_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.637026+0000","objects_scrubbed":5,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:05.922562+0000","scrub_duration":0.0094649629999999998,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2379,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973675+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:55.973675+0000","last_peered":"2024-09-16T13:30:55.973675+0000","last_clean":"2024-09-16T13:30:55.973675+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:55.973675+0000","last_undegraded":"2024-09-16T13:30:55.973675+0000","last_fullsized":"2024-09-16T13:30:55.973675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"145'422","reported_seq":4336,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973626+0000","last_change":"2024-09-16T13:30:32.936425+0000","last_active":"2024-09-16T13:30:55.973626+0000","last_peered":"2024-09-16T13:30:55.973626+0000","last_clean":"2024-09-16T13:30:55.973626+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:30:55.973626+0000","last_undegraded":"2024-09-16T13:30:55.973626+0000","last_fullsized":"2024-09-16T13:30:55.973626+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"144'407","reported_seq":3517,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972427+0000","last_change":"2024-09-16T13:30:32.940411+0000","last_active":"2024-09-16T13:30:55.972427+0000","last_peered":"2024-09-16T13:30:55.972427+0000","last_clean":"2024-09-16T13:30:55.972427+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:30:55.972427+0000","last_undegraded":"2024-09-16T13:30:55.972427+0000","last_fullsized":"2024-09-16T13:30:55.972427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"166'422","reported_seq":3595,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.770334+0000","last_change":"2024-09-16T13:31:10.770307+0000","last_active":"2024-09-16T13:31:10.770334+0000","last_peered":"2024-09-16T13:31:10.770334+0000","last_clean":"2024-09-16T13:31:10.770334+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:31:10.770334+0000","last_undegraded":"2024-09-16T13:31:10.770334+0000","last_fullsized":"2024-09-16T13:31:10.770334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'422","last_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_deep_scrub":"166'422","last_deep_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.770204+0000","objects_scrubbed":6,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:32.183077+0000","scrub_duration":0.0061710920000000004,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"143'412","reported_seq":3269,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972624+0000","last_change":"2024-09-16T13:30:32.942004+0000","last_active":"2024-09-16T13:30:55.972624+0000","last_peered":"2024-09-16T13:30:55.972624+0000","last_clean":"2024-09-16T13:30:55.972624+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:30:55.972624+0000","last_undegraded":"2024-09-16T13:30:55.972624+0000","last_fullsized":"2024-09-16T13:30:55.972624+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'661","reported_seq":2902,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.003871+0000","last_change":"2024-09-16T13:31:13.003826+0000","last_active":"2024-09-16T13:31:13.003871+0000","last_peered":"2024-09-16T13:31:13.003871+0000","last_clean":"2024-09-16T13:31:13.003871+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:31:13.003871+0000","last_undegraded":"2024-09-16T13:31:13.003871+0000","last_fullsized":"2024-09-16T13:31:13.003871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'661","last_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_deep_scrub":"166'661","last_deep_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.003719+0000","objects_scrubbed":7,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:28.799115+0000","scrub_duration":0.010013117,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"143'834","reported_seq":4138,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.207913+0000","last_change":"2024-09-16T13:30:51.198375+0000","last_active":"2024-09-16T13:30:54.207913+0000","last_peered":"2024-09-16T13:30:54.207913+0000","last_clean":"2024-09-16T13:30:54.207913+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:30:54.207913+0000","last_undegraded":"2024-09-16T13:30:54.207913+0000","last_fullsized":"2024-09-16T13:30:54.207913+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"166'499","reported_seq":3535,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.343908+0000","last_change":"2024-09-16T13:31:14.343867+0000","last_active":"2024-09-16T13:31:14.343908+0000","last_peered":"2024-09-16T13:31:14.343908+0000","last_clean":"2024-09-16T13:31:14.343908+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:31:14.343908+0000","last_undegraded":"2024-09-16T13:31:14.343908+0000","last_fullsized":"2024-09-16T13:31:14.343908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'499","last_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_deep_scrub":"166'499","last_deep_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.343753+0000","objects_scrubbed":10,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:54.979530+0000","scrub_duration":0.011985519,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'388","reported_seq":2046,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.669523+0000","last_change":"2024-09-16T13:31:04.669493+0000","last_active":"2024-09-16T13:31:04.669523+0000","last_peered":"2024-09-16T13:31:04.669523+0000","last_clean":"2024-09-16T13:31:04.669523+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:31:04.669523+0000","last_undegraded":"2024-09-16T13:31:04.669523+0000","last_fullsized":"2024-09-16T13:31:04.669523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'388","last_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_deep_scrub":"166'388","last_deep_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.669404+0000","objects_scrubbed":11,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:05.964062+0000","scrub_duration":0.010041550999999999,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'837","reported_seq":4384,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.260438+0000","last_change":"2024-09-16T13:31:14.260402+0000","last_active":"2024-09-16T13:31:14.260438+0000","last_peered":"2024-09-16T13:31:14.260438+0000","last_clean":"2024-09-16T13:31:14.260438+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:31:14.260438+0000","last_undegraded":"2024-09-16T13:31:14.260438+0000","last_fullsized":"2024-09-16T13:31:14.260438+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'837","last_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_deep_scrub":"166'837","last_deep_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.260308+0000","objects_scrubbed":10,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:12.589220+0000","scrub_duration":0.0088603329999999998,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"166'794","reported_seq":4888,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.408576+0000","last_change":"2024-09-16T13:31:05.408548+0000","last_active":"2024-09-16T13:31:05.408576+0000","last_peered":"2024-09-16T13:31:05.408576+0000","last_clean":"2024-09-16T13:31:05.408576+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:31:05.408576+0000","last_undegraded":"2024-09-16T13:31:05.408576+0000","last_fullsized":"2024-09-16T13:31:05.408576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'794","last_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_deep_scrub":"166'794","last_deep_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.408464+0000","objects_scrubbed":9,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:04.309754+0000","scrub_duration":0.0084311639999999997,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"143'762","reported_seq":3940,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.972454+0000","last_change":"2024-09-16T13:30:32.940382+0000","last_active":"2024-09-16T13:30:55.972454+0000","last_peered":"2024-09-16T13:30:55.972454+0000","last_clean":"2024-09-16T13:30:55.972454+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:30:55.972454+0000","last_undegraded":"2024-09-16T13:30:55.972454+0000","last_fullsized":"2024-09-16T13:30:55.972454+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":762,"log_dups_size":0,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.136136+0000","last_change":"2024-09-16T13:31:05.136109+0000","last_active":"2024-09-16T13:31:05.136136+0000","last_peered":"2024-09-16T13:31:05.136136+0000","last_clean":"2024-09-16T13:31:05.136136+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:31:05.136136+0000","last_undegraded":"2024-09-16T13:31:05.136136+0000","last_fullsized":"2024-09-16T13:31:05.136136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.136016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:13.629452+0000","scrub_duration":0.001343777,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"143'501","reported_seq":3623,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973564+0000","last_change":"2024-09-16T13:30:32.937270+0000","last_active":"2024-09-16T13:30:55.973564+0000","last_peered":"2024-09-16T13:30:55.973564+0000","last_clean":"2024-09-16T13:30:55.973564+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:30:55.973564+0000","last_undegraded":"2024-09-16T13:30:55.973564+0000","last_fullsized":"2024-09-16T13:30:55.973564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.198348+0000","last_change":"2024-09-16T13:31:02.198329+0000","last_active":"2024-09-16T13:31:02.198348+0000","last_peered":"2024-09-16T13:31:02.198348+0000","last_clean":"2024-09-16T13:31:02.198348+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:31:02.198348+0000","last_undegraded":"2024-09-16T13:31:02.198348+0000","last_fullsized":"2024-09-16T13:31:02.198348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.198261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:06.779026+0000","scrub_duration":0.0015109539999999999,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"166'371","reported_seq":2677,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.399969+0000","last_change":"2024-09-16T13:31:11.399927+0000","last_active":"2024-09-16T13:31:11.399969+0000","last_peered":"2024-09-16T13:31:11.399969+0000","last_clean":"2024-09-16T13:31:11.399969+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:31:11.399969+0000","last_undegraded":"2024-09-16T13:31:11.399969+0000","last_fullsized":"2024-09-16T13:31:11.399969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'371","last_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_deep_scrub":"166'371","last_deep_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.399838+0000","objects_scrubbed":11,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:03.104907+0000","scrub_duration":0.010935642000000001,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.157050+0000","last_change":"2024-09-16T13:31:14.157023+0000","last_active":"2024-09-16T13:31:14.157050+0000","last_peered":"2024-09-16T13:31:14.157050+0000","last_clean":"2024-09-16T13:31:14.157050+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:31:14.157050+0000","last_undegraded":"2024-09-16T13:31:14.157050+0000","last_fullsized":"2024-09-16T13:31:14.157050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.156924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:05:39.014384+0000","scrub_duration":0.001819235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"166'834","reported_seq":4271,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.992013+0000","last_change":"2024-09-16T13:31:04.991988+0000","last_active":"2024-09-16T13:31:04.992013+0000","last_peered":"2024-09-16T13:31:04.992013+0000","last_clean":"2024-09-16T13:31:04.992013+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:31:04.992013+0000","last_undegraded":"2024-09-16T13:31:04.992013+0000","last_fullsized":"2024-09-16T13:31:04.992013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'834","last_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_deep_scrub":"166'834","last_deep_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.991902+0000","objects_scrubbed":13,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:19.253050+0000","scrub_duration":0.01179998,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.164363+0000","last_change":"2024-09-16T13:31:05.164330+0000","last_active":"2024-09-16T13:31:05.164363+0000","last_peered":"2024-09-16T13:31:05.164363+0000","last_clean":"2024-09-16T13:31:05.164363+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:31:05.164363+0000","last_undegraded":"2024-09-16T13:31:05.164363+0000","last_fullsized":"2024-09-16T13:31:05.164363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.164235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:16.915040+0000","scrub_duration":0.0013633849999999999,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"166'595","reported_seq":3510,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.218310+0000","last_change":"2024-09-16T13:31:12.218250+0000","last_active":"2024-09-16T13:31:12.218310+0000","last_peered":"2024-09-16T13:31:12.218310+0000","last_clean":"2024-09-16T13:31:12.218310+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:31:12.218310+0000","last_undegraded":"2024-09-16T13:31:12.218310+0000","last_fullsized":"2024-09-16T13:31:12.218310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'595","last_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_deep_scrub":"166'595","last_deep_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.218143+0000","objects_scrubbed":2,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:42:49.903676+0000","scrub_duration":0.00655574,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:03.612635+0000","last_change":"2024-09-16T13:31:03.612609+0000","last_active":"2024-09-16T13:31:03.612635+0000","last_peered":"2024-09-16T13:31:03.612635+0000","last_clean":"2024-09-16T13:31:03.612635+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:31:03.612635+0000","last_undegraded":"2024-09-16T13:31:03.612635+0000","last_fullsized":"2024-09-16T13:31:03.612635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_clean_scrub_stamp":"2024-09-16T13:31:03.612520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:47.994981+0000","scrub_duration":0.0014637949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"166'3","reported_seq":125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.436058+0000","last_change":"2024-09-16T13:31:05.436033+0000","last_active":"2024-09-16T13:31:05.436058+0000","last_peered":"2024-09-16T13:31:05.436058+0000","last_clean":"2024-09-16T13:31:05.436058+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:31:05.436058+0000","last_undegraded":"2024-09-16T13:31:05.436058+0000","last_fullsized":"2024-09-16T13:31:05.436058+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'3","last_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_deep_scrub":"166'3","last_deep_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.435938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:04.127609+0000","scrub_duration":0.0034749310000000001,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"166'373","reported_seq":4581,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.203736+0000","last_change":"2024-09-16T13:31:10.203699+0000","last_active":"2024-09-16T13:31:10.203736+0000","last_peered":"2024-09-16T13:31:10.203736+0000","last_clean":"2024-09-16T13:31:10.203736+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:31:10.203736+0000","last_undegraded":"2024-09-16T13:31:10.203736+0000","last_fullsized":"2024-09-16T13:31:10.203736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'373","last_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_deep_scrub":"166'373","last_deep_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.203607+0000","objects_scrubbed":4,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:13.855585+0000","scrub_duration":0.00582097,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.132861+0000","last_change":"2024-09-16T13:31:06.132832+0000","last_active":"2024-09-16T13:31:06.132861+0000","last_peered":"2024-09-16T13:31:06.132861+0000","last_clean":"2024-09-16T13:31:06.132861+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:31:06.132861+0000","last_undegraded":"2024-09-16T13:31:06.132861+0000","last_fullsized":"2024-09-16T13:31:06.132861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.132727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:25:35.050519+0000","scrub_duration":0.001337249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.558177+0000","last_change":"2024-09-16T13:31:05.558149+0000","last_active":"2024-09-16T13:31:05.558177+0000","last_peered":"2024-09-16T13:31:05.558177+0000","last_clean":"2024-09-16T13:31:05.558177+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:31:05.558177+0000","last_undegraded":"2024-09-16T13:31:05.558177+0000","last_fullsized":"2024-09-16T13:31:05.558177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.558059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:07.215656+0000","scrub_duration":0.001425965,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"166'492","reported_seq":3488,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.228159+0000","last_change":"2024-09-16T13:31:13.228130+0000","last_active":"2024-09-16T13:31:13.228159+0000","last_peered":"2024-09-16T13:31:13.228159+0000","last_clean":"2024-09-16T13:31:13.228159+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:31:13.228159+0000","last_undegraded":"2024-09-16T13:31:13.228159+0000","last_fullsized":"2024-09-16T13:31:13.228159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'492","last_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_deep_scrub":"166'492","last_deep_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.228040+0000","objects_scrubbed":7,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:28.666805+0000","scrub_duration":0.0096771349999999999,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"166'6","reported_seq":165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.992554+0000","last_change":"2024-09-16T13:31:02.992512+0000","last_active":"2024-09-16T13:31:02.992554+0000","last_peered":"2024-09-16T13:31:02.992554+0000","last_clean":"2024-09-16T13:31:02.992554+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:31:02.992554+0000","last_undegraded":"2024-09-16T13:31:02.992554+0000","last_fullsized":"2024-09-16T13:31:02.992554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6","last_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_deep_scrub":"166'6","last_deep_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.992436+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:49.262037+0000","scrub_duration":0.0040630290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.419573+0000","last_change":"2024-09-16T13:31:05.419545+0000","last_active":"2024-09-16T13:31:05.419573+0000","last_peered":"2024-09-16T13:31:05.419573+0000","last_clean":"2024-09-16T13:31:05.419573+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:31:05.419573+0000","last_undegraded":"2024-09-16T13:31:05.419573+0000","last_fullsized":"2024-09-16T13:31:05.419573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.419455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:11.274470+0000","scrub_duration":0.0019415299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"166'612","reported_seq":4108,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.007306+0000","last_change":"2024-09-16T13:31:06.007280+0000","last_active":"2024-09-16T13:31:06.007306+0000","last_peered":"2024-09-16T13:31:06.007306+0000","last_clean":"2024-09-16T13:31:06.007306+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:31:06.007306+0000","last_undegraded":"2024-09-16T13:31:06.007306+0000","last_fullsized":"2024-09-16T13:31:06.007306+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'612","last_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_deep_scrub":"166'612","last_deep_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.007192+0000","objects_scrubbed":15,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:56.616376+0000","scrub_duration":0.011595329999999999,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"166'8","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.382630+0000","last_change":"2024-09-16T13:31:04.382593+0000","last_active":"2024-09-16T13:31:04.382630+0000","last_peered":"2024-09-16T13:31:04.382630+0000","last_clean":"2024-09-16T13:31:04.382630+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:31:04.382630+0000","last_undegraded":"2024-09-16T13:31:04.382630+0000","last_fullsized":"2024-09-16T13:31:04.382630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8","last_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_deep_scrub":"166'8","last_deep_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.382518+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:07:20.475579+0000","scrub_duration":0.0084622399999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"20'1","reported_seq":113,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:54.207826+0000","last_change":"2024-09-16T13:30:51.195846+0000","last_active":"2024-09-16T13:30:54.207826+0000","last_peered":"2024-09-16T13:30:54.207826+0000","last_clean":"2024-09-16T13:30:54.207826+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:30:54.207826+0000","last_undegraded":"2024-09-16T13:30:54.207826+0000","last_fullsized":"2024-09-16T13:30:54.207826+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:09.297020+0000","last_clean_scrub_stamp":"2024-09-16T13:28:09.297020+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"166'74","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.415903+0000","last_change":"2024-09-16T13:31:08.415862+0000","last_active":"2024-09-16T13:31:08.415903+0000","last_peered":"2024-09-16T13:31:08.415903+0000","last_clean":"2024-09-16T13:31:08.415903+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:31:08.415903+0000","last_undegraded":"2024-09-16T13:31:08.415903+0000","last_fullsized":"2024-09-16T13:31:08.415903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'74","last_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_deep_scrub":"166'74","last_deep_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.415767+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:17.920374+0000","scrub_duration":0.0052567780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"166'608","reported_seq":2536,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.165662+0000","last_change":"2024-09-16T13:31:11.165631+0000","last_active":"2024-09-16T13:31:11.165662+0000","last_peered":"2024-09-16T13:31:11.165662+0000","last_clean":"2024-09-16T13:31:11.165662+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:31:11.165662+0000","last_undegraded":"2024-09-16T13:31:11.165662+0000","last_fullsized":"2024-09-16T13:31:11.165662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'608","last_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_deep_scrub":"166'608","last_deep_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.165540+0000","objects_scrubbed":8,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:04:36.130808+0000","scrub_duration":0.0070728830000000003,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"166'1262","reported_seq":5761,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.458253+0000","last_change":"2024-09-16T13:31:16.458211+0000","last_active":"2024-09-16T13:31:16.458253+0000","last_peered":"2024-09-16T13:31:16.458253+0000","last_clean":"2024-09-16T13:31:16.458253+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:31:16.458253+0000","last_undegraded":"2024-09-16T13:31:16.458253+0000","last_fullsized":"2024-09-16T13:31:16.458253+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1262","last_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_deep_scrub":"166'1262","last_deep_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.458069+0000","objects_scrubbed":3,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:45.766858+0000","scrub_duration":0.0047390660000000001,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"144'398","reported_seq":4820,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973562+0000","last_change":"2024-09-16T13:30:32.936447+0000","last_active":"2024-09-16T13:30:55.973562+0000","last_peered":"2024-09-16T13:30:55.973562+0000","last_clean":"2024-09-16T13:30:55.973562+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:30:55.973562+0000","last_undegraded":"2024-09-16T13:30:55.973562+0000","last_fullsized":"2024-09-16T13:30:55.973562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"166'582","reported_seq":4619,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.179053+0000","last_change":"2024-09-16T13:31:13.179007+0000","last_active":"2024-09-16T13:31:13.179053+0000","last_peered":"2024-09-16T13:31:13.179053+0000","last_clean":"2024-09-16T13:31:13.179053+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:31:13.179053+0000","last_undegraded":"2024-09-16T13:31:13.179053+0000","last_fullsized":"2024-09-16T13:31:13.179053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'582","last_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_deep_scrub":"166'582","last_deep_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.178913+0000","objects_scrubbed":10,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:45.723440+0000","scrub_duration":0.0078661749999999996,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"166'434","reported_seq":5531,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:18.125960+0000","last_change":"2024-09-16T13:31:18.125928+0000","last_active":"2024-09-16T13:31:18.125960+0000","last_peered":"2024-09-16T13:31:18.125960+0000","last_clean":"2024-09-16T13:31:18.125960+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:31:18.125960+0000","last_undegraded":"2024-09-16T13:31:18.125960+0000","last_fullsized":"2024-09-16T13:31:18.125960+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'434","last_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_deep_scrub":"166'434","last_deep_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_clean_scrub_stamp":"2024-09-16T13:31:18.125809+0000","objects_scrubbed":9,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:49.862293+0000","scrub_duration":0.0072992939999999996,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"147'512","reported_seq":5589,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973737+0000","last_change":"2024-09-16T13:30:32.937587+0000","last_active":"2024-09-16T13:30:55.973737+0000","last_peered":"2024-09-16T13:30:55.973737+0000","last_clean":"2024-09-16T13:30:55.973737+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:30:55.973737+0000","last_undegraded":"2024-09-16T13:30:55.973737+0000","last_fullsized":"2024-09-16T13:30:55.973737+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"166'469","reported_seq":3192,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.456588+0000","last_change":"2024-09-16T13:31:15.456540+0000","last_active":"2024-09-16T13:31:15.456588+0000","last_peered":"2024-09-16T13:31:15.456588+0000","last_clean":"2024-09-16T13:31:15.456588+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:31:15.456588+0000","last_undegraded":"2024-09-16T13:31:15.456588+0000","last_fullsized":"2024-09-16T13:31:15.456588+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'469","last_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_deep_scrub":"166'469","last_deep_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.456434+0000","objects_scrubbed":12,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:21:06.510924+0000","scrub_duration":0.010685587999999999,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"166'382","reported_seq":1833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.868714+0000","last_change":"2024-09-16T13:31:12.868676+0000","last_active":"2024-09-16T13:31:12.868714+0000","last_peered":"2024-09-16T13:31:12.868714+0000","last_clean":"2024-09-16T13:31:12.868714+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:31:12.868714+0000","last_undegraded":"2024-09-16T13:31:12.868714+0000","last_fullsized":"2024-09-16T13:31:12.868714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'382","last_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_deep_scrub":"166'382","last_deep_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.868550+0000","objects_scrubbed":17,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:10.566793+0000","scrub_duration":0.012531097999999999,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"166'859","reported_seq":3466,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.652501+0000","last_change":"2024-09-16T13:31:15.652472+0000","last_active":"2024-09-16T13:31:15.652501+0000","last_peered":"2024-09-16T13:31:15.652501+0000","last_clean":"2024-09-16T13:31:15.652501+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:31:15.652501+0000","last_undegraded":"2024-09-16T13:31:15.652501+0000","last_fullsized":"2024-09-16T13:31:15.652501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'859","last_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_deep_scrub":"166'859","last_deep_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.652383+0000","objects_scrubbed":11,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:42.495261+0000","scrub_duration":0.010275467,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18645,"ondisk_log_size":18645,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509484,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303496,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068344,"statfs":{"total":96636764160,"available":96325984256,"internally_reserved":0,"allocated":2031616,"data_stored":1606469,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46200000000000002,"15min":0.47599999999999998},"min":{"1min":0.32300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46400000000000002,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.192,"15min":0.192},"max":{"1min":0.64800000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.61499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67700000000000005,"5min":0.879,"15min":0.879},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59699999999999998,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67600000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38600000000000001,"15min":0.40400000000000003},"min":{"1min":0.23999999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.439}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60499999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65200000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.53400000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509486,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303468,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068372,"statfs":{"total":96636764160,"available":96326012928,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.34699999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0029999999999999,"5min":2.169,"15min":2.169},"last":0.44}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.45300000000000001,"15min":0.45200000000000001},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.5},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.39500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.68400000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55300000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.54000000000000004},"min":{"1min":0.33700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.93600000000000005,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.496,"15min":0.496},"min":{"1min":0.371,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40400000000000003,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.64800000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64300000000000002,"5min":0.752,"15min":0.752},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44400000000000001,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.31},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.495},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.47599999999999998},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.29799999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.877,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.35499999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509485,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303468,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068372,"statfs":{"total":96636764160,"available":96326012928,"internally_reserved":0,"allocated":2002944,"data_stored":1016286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.71599999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68799999999999994,"5min":1.173,"15min":1.173},"last":0.626}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.501,"15min":0.51700000000000002},"min":{"1min":0.36799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70099999999999996,"5min":1.224,"15min":1.224},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64600000000000002,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.38400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63600000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.49399999999999999},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66700000000000004,"5min":1.296,"15min":1.296},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.67100000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.443,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64600000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68899999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.499},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46600000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68300000000000005,"5min":0.747,"15min":0.747},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72299999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.66700000000000004}]}]},{"osd":3,"up_from":15,"seq":64424509484,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307628,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064212,"statfs":{"total":96636764160,"available":96321753088,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.38100000000000001},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35699999999999998,"15min":0.376},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.59299999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36399999999999999,"15min":0.375},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.504,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.34599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":1.093,"15min":1.093},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.38200000000000001,"15min":0.39000000000000001},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.47699999999999998,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.37}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.71599999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70999999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Sep 16 13:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70399999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68999999999999995,"5min":1.119,"15min":1.119},"last":0.45300000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542191,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299296,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072544,"statfs":{"total":96636764160,"available":96330285056,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.66700000000000004,"15min":0.75600000000000001},"min":{"1min":0.34799999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.67400000000000004,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.498,"15min":0.496},"min":{"1min":0.39300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.55100000000000005},"min":{"1min":0.36899999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":1.286,"15min":1.286},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71799999999999997,"5min":1.236,"15min":1.236},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.371,"5min":0.245,"15min":0.245},"max":{"1min":0.61399999999999999,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64900000000000002,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.58199999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":1.288,"15min":1.288},"last":0.48699999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68999999999999995,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.41999999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47499999999999998,"15min":0.503},"min":{"1min":0.314,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70699999999999996,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72699999999999998,"5min":1.008,"15min":1.008},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.377},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47199999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307616,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064224,"statfs":{"total":96636764160,"available":96321765376,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33300000000000002,"15min":0.34300000000000003},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.50600000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.60499999999999998,"15min":0.72499999999999998},"min":{"1min":0.26100000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.47499999999999998,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34399999999999997,"15min":0.34999999999999998},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.60399999999999998,"15min":0.72199999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.47999999999999998,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38800000000000001,"15min":0.40200000000000002},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46300000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.28199999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.53700000000000003,"5min":1.194,"15min":1.194},"last":0.53500000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54800000000000004,"15min":0.57099999999999995},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79000000000000004,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67100000000000004,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.72199999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.751,"5min":1.321,"15min":1.321},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.66000000000000003,"5min":1.23,"15min":1.23},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.77700000000000002,"15min":0.88500000000000001},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69699999999999995,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77200000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.79100000000000004,"15min":0.90200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67600000000000005,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72799999999999998,"5min":1.347,"15min":1.347},"last":0.70599999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542191,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1994752,"data_stored":1016120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41799999999999998,"15min":0.371},"min":{"1min":0.216,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.34000000000000002},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55600000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.42799999999999999,"15min":0.38},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.58899999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.248,"5min":0.221,"15min":0.221},"max":{"1min":0.50900000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.46700000000000003,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.441}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.37,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63700000000000001,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80100000000000005,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.64100000000000001,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.38,"15min":0.38},"max":{"1min":0.871,"5min":8.298,"15min":8.298},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.40999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":1.181,"15min":1.181},"last":0.70499999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542189,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299360,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072480,"statfs":{"total":96636764160,"available":96330219520,"internally_reserved":0,"allocated":2056192,"data_stored":1606635,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.442,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.35799999999999998,"15min":0.36399999999999999},"min":{"1min":0.29099999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33900000000000002},"min":{"1min":0.224,"5min":0.192,"15min":0.192},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65100000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.377,"15min":0.379},"min":{"1min":0.255,"5min":0.249,"15min":0.249},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53500000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71099999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79400000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.623},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.56100000000000005},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78600000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81200000000000006,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78000000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.228 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.229 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.229 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.229 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T13:28:09.297020+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.229 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.229 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:24.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:31:44.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:31:44.503 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:31:44.504 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:31:44.514 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-16T13:31:43.879943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590742,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21783,"num_write_kb":462549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18833,"ondisk_log_size":18833,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2440272,"kb_used_data":15696,"kb_used_omap":13,"kb_used_meta":2424498,"kb_avail":752534448,"statfs":{"total":773094113280,"available":770595274752,"internally_reserved":0,"allocated":16072704,"data_stored":9310292,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2482686656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002636"},"pg_stats":[{"pgid":"4.1f","version":"166'378","reported_seq":2287,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.175418+0000","last_change":"2024-09-16T13:31:08.175392+0000","last_active":"2024-09-16T13:31:08.175418+0000","last_peered":"2024-09-16T13:31:08.175418+0000","last_clean":"2024-09-16T13:31:08.175418+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:31:08.175418+0000","last_undegraded":"2024-09-16T13:31:08.175418+0000","last_fullsized":"2024-09-16T13:31:08.175418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'378","last_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_deep_scrub":"166'378","last_deep_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.175307+0000","objects_scrubbed":11,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:49.299655+0000","scrub_duration":0.010270289,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"166'1105","reported_seq":11867,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.010295+0000","last_change":"2024-09-16T13:31:16.010249+0000","last_active":"2024-09-16T13:31:16.010295+0000","last_peered":"2024-09-16T13:31:16.010295+0000","last_clean":"2024-09-16T13:31:16.010295+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:31:16.010295+0000","last_undegraded":"2024-09-16T13:31:16.010295+0000","last_fullsized":"2024-09-16T13:31:16.010295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1105","last_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_deep_scrub":"166'1105","last_deep_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.010149+0000","objects_scrubbed":10,"log_size":1105,"log_dups_size":0,"ondisk_log_size":1105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:32.387613+0000","scrub_duration":0.009411081,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"166'458","reported_seq":1702,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:30.418020+0000","last_change":"2024-09-16T13:31:30.417988+0000","last_active":"2024-09-16T13:31:30.418020+0000","last_peered":"2024-09-16T13:31:30.418020+0000","last_clean":"2024-09-16T13:31:30.418020+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:31:30.418020+0000","last_undegraded":"2024-09-16T13:31:30.418020+0000","last_fullsized":"2024-09-16T13:31:30.418020+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'458","last_scrub_stamp":"2024-09-16T13:31:30.417882+0000","last_deep_scrub":"166'458","last_deep_scrub_stamp":"2024-09-16T13:31:30.417882+0000","last_clean_scrub_stamp":"2024-09-16T13:31:30.417882+0000","objects_scrubbed":10,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:12:24.945508+0000","scrub_duration":0.0087942999999999997,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"166'515","reported_seq":4617,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.637148+0000","last_change":"2024-09-16T13:31:13.637117+0000","last_active":"2024-09-16T13:31:13.637148+0000","last_peered":"2024-09-16T13:31:13.637148+0000","last_clean":"2024-09-16T13:31:13.637148+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:31:13.637148+0000","last_undegraded":"2024-09-16T13:31:13.637148+0000","last_fullsized":"2024-09-16T13:31:13.637148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'515","last_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_deep_scrub":"166'515","last_deep_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.637026+0000","objects_scrubbed":5,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:05.922562+0000","scrub_duration":0.0094649629999999998,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"141'477","reported_seq":2379,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:30:55.973675+0000","last_change":"2024-09-16T13:30:32.936806+0000","last_active":"2024-09-16T13:30:55.973675+0000","last_peered":"2024-09-16T13:30:55.973675+0000","last_clean":"2024-09-16T13:30:55.973675+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:30:55.973675+0000","last_undegraded":"2024-09-16T13:30:55.973675+0000","last_fullsized":"2024-09-16T13:30:55.973675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:28:25.959435+0000","last_clean_scrub_stamp":"2024-09-16T13:28:25.959435+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"166'433","reported_seq":4357,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:26.910273+0000","last_change":"2024-09-16T13:31:26.910245+0000","last_active":"2024-09-16T13:31:26.910273+0000","last_peered":"2024-09-16T13:31:26.910273+0000","last_clean":"2024-09-16T13:31:26.910273+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:31:26.910273+0000","last_undegraded":"2024-09-16T13:31:26.910273+0000","last_fullsized":"2024-09-16T13:31:26.910273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'433","last_scrub_stamp":"2024-09-16T13:31:26.910152+0000","last_deep_scrub":"166'433","last_deep_scrub_stamp":"2024-09-16T13:31:26.910152+0000","last_clean_scrub_stamp":"2024-09-16T13:31:26.910152+0000","objects_scrubbed":11,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:39.628602+0000","scrub_duration":0.0084123670000000005,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"166'417","reported_seq":3537,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:24.386457+0000","last_change":"2024-09-16T13:31:24.386420+0000","last_active":"2024-09-16T13:31:24.386457+0000","last_peered":"2024-09-16T13:31:24.386457+0000","last_clean":"2024-09-16T13:31:24.386457+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:31:24.386457+0000","last_undegraded":"2024-09-16T13:31:24.386457+0000","last_fullsized":"2024-09-16T13:31:24.386457+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'417","last_scrub_stamp":"2024-09-16T13:31:24.386338+0000","last_deep_scrub":"166'417","last_deep_scrub_stamp":"2024-09-16T13:31:24.386338+0000","last_clean_scrub_stamp":"2024-09-16T13:31:24.386338+0000","objects_scrubbed":10,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:51.226800+0000","scrub_duration":0.0067827499999999997,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"166'422","reported_seq":3595,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.770334+0000","last_change":"2024-09-16T13:31:10.770307+0000","last_active":"2024-09-16T13:31:10.770334+0000","last_peered":"2024-09-16T13:31:10.770334+0000","last_clean":"2024-09-16T13:31:10.770334+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:31:10.770334+0000","last_undegraded":"2024-09-16T13:31:10.770334+0000","last_fullsized":"2024-09-16T13:31:10.770334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'422","last_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_deep_scrub":"166'422","last_deep_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.770204+0000","objects_scrubbed":6,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:32.183077+0000","scrub_duration":0.0061710920000000004,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"166'427","reported_seq":3294,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:22.399595+0000","last_change":"2024-09-16T13:31:22.399560+0000","last_active":"2024-09-16T13:31:22.399595+0000","last_peered":"2024-09-16T13:31:22.399595+0000","last_clean":"2024-09-16T13:31:22.399595+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:31:22.399595+0000","last_undegraded":"2024-09-16T13:31:22.399595+0000","last_fullsized":"2024-09-16T13:31:22.399595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'427","last_scrub_stamp":"2024-09-16T13:31:22.399465+0000","last_deep_scrub":"166'427","last_deep_scrub_stamp":"2024-09-16T13:31:22.399465+0000","last_clean_scrub_stamp":"2024-09-16T13:31:22.399465+0000","objects_scrubbed":15,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:37.784900+0000","scrub_duration":0.011867277000000001,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'661","reported_seq":2902,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.003871+0000","last_change":"2024-09-16T13:31:13.003826+0000","last_active":"2024-09-16T13:31:13.003871+0000","last_peered":"2024-09-16T13:31:13.003871+0000","last_clean":"2024-09-16T13:31:13.003871+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:31:13.003871+0000","last_undegraded":"2024-09-16T13:31:13.003871+0000","last_fullsized":"2024-09-16T13:31:13.003871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'661","last_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_deep_scrub":"166'661","last_deep_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.003719+0000","objects_scrubbed":7,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:28.799115+0000","scrub_duration":0.010013117,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"166'843","reported_seq":4157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:22.466996+0000","last_change":"2024-09-16T13:31:22.466960+0000","last_active":"2024-09-16T13:31:22.466996+0000","last_peered":"2024-09-16T13:31:22.466996+0000","last_clean":"2024-09-16T13:31:22.466996+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:31:22.466996+0000","last_undegraded":"2024-09-16T13:31:22.466996+0000","last_fullsized":"2024-09-16T13:31:22.466996+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'843","last_scrub_stamp":"2024-09-16T13:31:22.466858+0000","last_deep_scrub":"166'843","last_deep_scrub_stamp":"2024-09-16T13:31:22.466858+0000","last_clean_scrub_stamp":"2024-09-16T13:31:22.466858+0000","objects_scrubbed":9,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:55.207634+0000","scrub_duration":0.0084471040000000004,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"166'499","reported_seq":3535,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.343908+0000","last_change":"2024-09-16T13:31:14.343867+0000","last_active":"2024-09-16T13:31:14.343908+0000","last_peered":"2024-09-16T13:31:14.343908+0000","last_clean":"2024-09-16T13:31:14.343908+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:31:14.343908+0000","last_undegraded":"2024-09-16T13:31:14.343908+0000","last_fullsized":"2024-09-16T13:31:14.343908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'499","last_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_deep_scrub":"166'499","last_deep_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.343753+0000","objects_scrubbed":10,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:54.979530+0000","scrub_duration":0.011985519,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'388","reported_seq":2046,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.669523+0000","last_change":"2024-09-16T13:31:04.669493+0000","last_active":"2024-09-16T13:31:04.669523+0000","last_peered":"2024-09-16T13:31:04.669523+0000","last_clean":"2024-09-16T13:31:04.669523+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:31:04.669523+0000","last_undegraded":"2024-09-16T13:31:04.669523+0000","last_fullsized":"2024-09-16T13:31:04.669523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'388","last_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_deep_scrub":"166'388","last_deep_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.669404+0000","objects_scrubbed":11,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:05.964062+0000","scrub_duration":0.010041550999999999,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'837","reported_seq":4384,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.260438+0000","last_change":"2024-09-16T13:31:14.260402+0000","last_active":"2024-09-16T13:31:14.260438+0000","last_peered":"2024-09-16T13:31:14.260438+0000","last_clean":"2024-09-16T13:31:14.260438+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:31:14.260438+0000","last_undegraded":"2024-09-16T13:31:14.260438+0000","last_fullsized":"2024-09-16T13:31:14.260438+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'837","last_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_deep_scrub":"166'837","last_deep_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.260308+0000","objects_scrubbed":10,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:12.589220+0000","scrub_duration":0.0088603329999999998,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"166'794","reported_seq":4888,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.408576+0000","last_change":"2024-09-16T13:31:05.408548+0000","last_active":"2024-09-16T13:31:05.408576+0000","last_peered":"2024-09-16T13:31:05.408576+0000","last_clean":"2024-09-16T13:31:05.408576+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:31:05.408576+0000","last_undegraded":"2024-09-16T13:31:05.408576+0000","last_fullsized":"2024-09-16T13:31:05.408576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'794","last_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_deep_scrub":"166'794","last_deep_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.408464+0000","objects_scrubbed":9,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:04.309754+0000","scrub_duration":0.0084311639999999997,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"166'772","reported_seq":3960,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:19.399335+0000","last_change":"2024-09-16T13:31:19.399306+0000","last_active":"2024-09-16T13:31:19.399335+0000","last_peered":"2024-09-16T13:31:19.399335+0000","last_clean":"2024-09-16T13:31:19.399335+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:31:19.399335+0000","last_undegraded":"2024-09-16T13:31:19.399335+0000","last_fullsized":"2024-09-16T13:31:19.399335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'772","last_scrub_stamp":"2024-09-16T13:31:19.399215+0000","last_deep_scrub":"166'772","last_deep_scrub_stamp":"2024-09-16T13:31:19.399215+0000","last_clean_scrub_stamp":"2024-09-16T13:31:19.399215+0000","objects_scrubbed":10,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:34.493382+0000","scrub_duration":0.010399172999999999,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.136136+0000","last_change":"2024-09-16T13:31:05.136109+0000","last_active":"2024-09-16T13:31:05.136136+0000","last_peered":"2024-09-16T13:31:05.136136+0000","last_clean":"2024-09-16T13:31:05.136136+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:31:05.136136+0000","last_undegraded":"2024-09-16T13:31:05.136136+0000","last_fullsized":"2024-09-16T13:31:05.136136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.136016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:13.629452+0000","scrub_duration":0.001343777,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"166'510","reported_seq":3642,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:27.873288+0000","last_change":"2024-09-16T13:31:27.873250+0000","last_active":"2024-09-16T13:31:27.873288+0000","last_peered":"2024-09-16T13:31:27.873288+0000","last_clean":"2024-09-16T13:31:27.873288+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:31:27.873288+0000","last_undegraded":"2024-09-16T13:31:27.873288+0000","last_fullsized":"2024-09-16T13:31:27.873288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'510","last_scrub_stamp":"2024-09-16T13:31:27.873151+0000","last_deep_scrub":"166'510","last_deep_scrub_stamp":"2024-09-16T13:31:27.873151+0000","last_clean_scrub_stamp":"2024-09-16T13:31:27.873151+0000","objects_scrubbed":9,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:30:24.359148+0000","scrub_duration":0.010434084999999999,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.198348+0000","last_change":"2024-09-16T13:31:02.198329+0000","last_active":"2024-09-16T13:31:02.198348+0000","last_peered":"2024-09-16T13:31:02.198348+0000","last_clean":"2024-09-16T13:31:02.198348+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:31:02.198348+0000","last_undegraded":"2024-09-16T13:31:02.198348+0000","last_fullsized":"2024-09-16T13:31:02.198348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.198261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:06.779026+0000","scrub_duration":0.0015109539999999999,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"166'371","reported_seq":2677,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.399969+0000","last_change":"2024-09-16T13:31:11.399927+0000","last_active":"2024-09-16T13:31:11.399969+0000","last_peered":"2024-09-16T13:31:11.399969+0000","last_clean":"2024-09-16T13:31:11.399969+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:31:11.399969+0000","last_undegraded":"2024-09-16T13:31:11.399969+0000","last_fullsized":"2024-09-16T13:31:11.399969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'371","last_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_deep_scrub":"166'371","last_deep_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.399838+0000","objects_scrubbed":11,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:03.104907+0000","scrub_duration":0.010935642000000001,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.157050+0000","last_change":"2024-09-16T13:31:14.157023+0000","last_active":"2024-09-16T13:31:14.157050+0000","last_peered":"2024-09-16T13:31:14.157050+0000","last_clean":"2024-09-16T13:31:14.157050+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:31:14.157050+0000","last_undegraded":"2024-09-16T13:31:14.157050+0000","last_fullsized":"2024-09-16T13:31:14.157050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.156924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:05:39.014384+0000","scrub_duration":0.001819235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"166'834","reported_seq":4271,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.992013+0000","last_change":"2024-09-16T13:31:04.991988+0000","last_active":"2024-09-16T13:31:04.992013+0000","last_peered":"2024-09-16T13:31:04.992013+0000","last_clean":"2024-09-16T13:31:04.992013+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:31:04.992013+0000","last_undegraded":"2024-09-16T13:31:04.992013+0000","last_fullsized":"2024-09-16T13:31:04.992013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'834","last_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_deep_scrub":"166'834","last_deep_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.991902+0000","objects_scrubbed":13,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:19.253050+0000","scrub_duration":0.01179998,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.164363+0000","last_change":"2024-09-16T13:31:05.164330+0000","last_active":"2024-09-16T13:31:05.164363+0000","last_peered":"2024-09-16T13:31:05.164363+0000","last_clean":"2024-09-16T13:31:05.164363+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:31:05.164363+0000","last_undegraded":"2024-09-16T13:31:05.164363+0000","last_fullsized":"2024-09-16T13:31:05.164363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.164235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:16.915040+0000","scrub_duration":0.0013633849999999999,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"166'595","reported_seq":3510,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.218310+0000","last_change":"2024-09-16T13:31:12.218250+0000","last_active":"2024-09-16T13:31:12.218310+0000","last_peered":"2024-09-16T13:31:12.218310+0000","last_clean":"2024-09-16T13:31:12.218310+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:31:12.218310+0000","last_undegraded":"2024-09-16T13:31:12.218310+0000","last_fullsized":"2024-09-16T13:31:12.218310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'595","last_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_deep_scrub":"166'595","last_deep_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.218143+0000","objects_scrubbed":2,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:42:49.903676+0000","scrub_duration":0.00655574,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:03.612635+0000","last_change":"2024-09-16T13:31:03.612609+0000","last_active":"2024-09-16T13:31:03.612635+0000","last_peered":"2024-09-16T13:31:03.612635+0000","last_clean":"2024-09-16T13:31:03.612635+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:31:03.612635+0000","last_undegraded":"2024-09-16T13:31:03.612635+0000","last_fullsized":"2024-09-16T13:31:03.612635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_clean_scrub_stamp":"2024-09-16T13:31:03.612520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:47.994981+0000","scrub_duration":0.0014637949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"166'3","reported_seq":125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.436058+0000","last_change":"2024-09-16T13:31:05.436033+0000","last_active":"2024-09-16T13:31:05.436058+0000","last_peered":"2024-09-16T13:31:05.436058+0000","last_clean":"2024-09-16T13:31:05.436058+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:31:05.436058+0000","last_undegraded":"2024-09-16T13:31:05.436058+0000","last_fullsized":"2024-09-16T13:31:05.436058+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'3","last_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_deep_scrub":"166'3","last_deep_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.435938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:04.127609+0000","scrub_duration":0.0034749310000000001,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"166'373","reported_seq":4581,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.203736+0000","last_change":"2024-09-16T13:31:10.203699+0000","last_active":"2024-09-16T13:31:10.203736+0000","last_peered":"2024-09-16T13:31:10.203736+0000","last_clean":"2024-09-16T13:31:10.203736+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:31:10.203736+0000","last_undegraded":"2024-09-16T13:31:10.203736+0000","last_fullsized":"2024-09-16T13:31:10.203736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'373","last_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_deep_scrub":"166'373","last_deep_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.203607+0000","objects_scrubbed":4,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:13.855585+0000","scrub_duration":0.00582097,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.132861+0000","last_change":"2024-09-16T13:31:06.132832+0000","last_active":"2024-09-16T13:31:06.132861+0000","last_peered":"2024-09-16T13:31:06.132861+0000","last_clean":"2024-09-16T13:31:06.132861+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:31:06.132861+0000","last_undegraded":"2024-09-16T13:31:06.132861+0000","last_fullsized":"2024-09-16T13:31:06.132861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.132727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:25:35.050519+0000","scrub_duration":0.001337249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.558177+0000","last_change":"2024-09-16T13:31:05.558149+0000","last_active":"2024-09-16T13:31:05.558177+0000","last_peered":"2024-09-16T13:31:05.558177+0000","last_clean":"2024-09-16T13:31:05.558177+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:31:05.558177+0000","last_undegraded":"2024-09-16T13:31:05.558177+0000","last_fullsized":"2024-09-16T13:31:05.558177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.558059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:07.215656+0000","scrub_duration":0.001425965,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"166'492","reported_seq":3488,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.228159+0000","last_change":"2024-09-16T13:31:13.228130+0000","last_active":"2024-09-16T13:31:13.228159+0000","last_peered":"2024-09-16T13:31:13.228159+0000","last_clean":"2024-09-16T13:31:13.228159+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:31:13.228159+0000","last_undegraded":"2024-09-16T13:31:13.228159+0000","last_fullsized":"2024-09-16T13:31:13.228159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'492","last_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_deep_scrub":"166'492","last_deep_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.228040+0000","objects_scrubbed":7,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:28.666805+0000","scrub_duration":0.0096771349999999999,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"166'6","reported_seq":165,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.992554+0000","last_change":"2024-09-16T13:31:02.992512+0000","last_active":"2024-09-16T13:31:02.992554+0000","last_peered":"2024-09-16T13:31:02.992554+0000","last_clean":"2024-09-16T13:31:02.992554+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:31:02.992554+0000","last_undegraded":"2024-09-16T13:31:02.992554+0000","last_fullsized":"2024-09-16T13:31:02.992554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6","last_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_deep_scrub":"166'6","last_deep_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.992436+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:49.262037+0000","scrub_duration":0.0040630290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.419573+0000","last_change":"2024-09-16T13:31:05.419545+0000","last_active":"2024-09-16T13:31:05.419573+0000","last_peered":"2024-09-16T13:31:05.419573+0000","last_clean":"2024-09-16T13:31:05.419573+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:31:05.419573+0000","last_undegraded":"2024-09-16T13:31:05.419573+0000","last_fullsized":"2024-09-16T13:31:05.419573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.419455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:11.274470+0000","scrub_duration":0.0019415299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"166'612","reported_seq":4108,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.007306+0000","last_change":"2024-09-16T13:31:06.007280+0000","last_active":"2024-09-16T13:31:06.007306+0000","last_peered":"2024-09-16T13:31:06.007306+0000","last_clean":"2024-09-16T13:31:06.007306+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:31:06.007306+0000","last_undegraded":"2024-09-16T13:31:06.007306+0000","last_fullsized":"2024-09-16T13:31:06.007306+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'612","last_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_deep_scrub":"166'612","last_deep_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.007192+0000","objects_scrubbed":15,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:56.616376+0000","scrub_duration":0.011595329999999999,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"166'8","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.382630+0000","last_change":"2024-09-16T13:31:04.382593+0000","last_active":"2024-09-16T13:31:04.382630+0000","last_peered":"2024-09-16T13:31:04.382630+0000","last_clean":"2024-09-16T13:31:04.382630+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:31:04.382630+0000","last_undegraded":"2024-09-16T13:31:04.382630+0000","last_fullsized":"2024-09-16T13:31:04.382630+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8","last_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_deep_scrub":"166'8","last_deep_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.382518+0000","objects_scrubbed":4,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:07:20.475579+0000","scrub_duration":0.0084622399999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":170,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"166'2","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:19.461782+0000","last_change":"2024-09-16T13:31:19.461736+0000","last_active":"2024-09-16T13:31:19.461782+0000","last_peered":"2024-09-16T13:31:19.461782+0000","last_clean":"2024-09-16T13:31:19.461782+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:31:19.461782+0000","last_undegraded":"2024-09-16T13:31:19.461782+0000","last_fullsized":"2024-09-16T13:31:19.461782+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-09-16T13:31:19.461633+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-09-16T13:31:19.461633+0000","last_clean_scrub_stamp":"2024-09-16T13:31:19.461633+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:49.388690+0000","scrub_duration":0.003386072,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"166'74","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.415903+0000","last_change":"2024-09-16T13:31:08.415862+0000","last_active":"2024-09-16T13:31:08.415903+0000","last_peered":"2024-09-16T13:31:08.415903+0000","last_clean":"2024-09-16T13:31:08.415903+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:31:08.415903+0000","last_undegraded":"2024-09-16T13:31:08.415903+0000","last_fullsized":"2024-09-16T13:31:08.415903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'74","last_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_deep_scrub":"166'74","last_deep_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.415767+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:17.920374+0000","scrub_duration":0.0052567780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"166'608","reported_seq":2536,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.165662+0000","last_change":"2024-09-16T13:31:11.165631+0000","last_active":"2024-09-16T13:31:11.165662+0000","last_peered":"2024-09-16T13:31:11.165662+0000","last_clean":"2024-09-16T13:31:11.165662+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:31:11.165662+0000","last_undegraded":"2024-09-16T13:31:11.165662+0000","last_fullsized":"2024-09-16T13:31:11.165662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'608","last_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_deep_scrub":"166'608","last_deep_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.165540+0000","objects_scrubbed":8,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:04:36.130808+0000","scrub_duration":0.0070728830000000003,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"166'1262","reported_seq":5761,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.458253+0000","last_change":"2024-09-16T13:31:16.458211+0000","last_active":"2024-09-16T13:31:16.458253+0000","last_peered":"2024-09-16T13:31:16.458253+0000","last_clean":"2024-09-16T13:31:16.458253+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:31:16.458253+0000","last_undegraded":"2024-09-16T13:31:16.458253+0000","last_fullsized":"2024-09-16T13:31:16.458253+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1262","last_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_deep_scrub":"166'1262","last_deep_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.458069+0000","objects_scrubbed":3,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:45.766858+0000","scrub_duration":0.0047390660000000001,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"166'403","reported_seq":4835,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:34.792864+0000","last_change":"2024-09-16T13:31:34.792827+0000","last_active":"2024-09-16T13:31:34.792864+0000","last_peered":"2024-09-16T13:31:34.792864+0000","last_clean":"2024-09-16T13:31:34.792864+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:31:34.792864+0000","last_undegraded":"2024-09-16T13:31:34.792864+0000","last_fullsized":"2024-09-16T13:31:34.792864+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'403","last_scrub_stamp":"2024-09-16T13:31:34.792734+0000","last_deep_scrub":"166'403","last_deep_scrub_stamp":"2024-09-16T13:31:34.792734+0000","last_clean_scrub_stamp":"2024-09-16T13:31:34.792734+0000","objects_scrubbed":5,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:52:15.651803+0000","scrub_duration":0.0071034569999999997,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"166'582","reported_seq":4619,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.179053+0000","last_change":"2024-09-16T13:31:13.179007+0000","last_active":"2024-09-16T13:31:13.179053+0000","last_peered":"2024-09-16T13:31:13.179053+0000","last_clean":"2024-09-16T13:31:13.179053+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:31:13.179053+0000","last_undegraded":"2024-09-16T13:31:13.179053+0000","last_fullsized":"2024-09-16T13:31:13.179053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'582","last_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_deep_scrub":"166'582","last_deep_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.178913+0000","objects_scrubbed":10,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:45.723440+0000","scrub_duration":0.0078661749999999996,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"166'434","reported_seq":5531,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:18.125960+0000","last_change":"2024-09-16T13:31:18.125928+0000","last_active":"2024-09-16T13:31:18.125960+0000","last_peered":"2024-09-16T13:31:18.125960+0000","last_clean":"2024-09-16T13:31:18.125960+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:31:18.125960+0000","last_undegraded":"2024-09-16T13:31:18.125960+0000","last_fullsized":"2024-09-16T13:31:18.125960+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'434","last_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_deep_scrub":"166'434","last_deep_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_clean_scrub_stamp":"2024-09-16T13:31:18.125809+0000","objects_scrubbed":9,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:49.862293+0000","scrub_duration":0.0072992939999999996,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"166'528","reported_seq":5615,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:31.799202+0000","last_change":"2024-09-16T13:31:31.799176+0000","last_active":"2024-09-16T13:31:31.799202+0000","last_peered":"2024-09-16T13:31:31.799202+0000","last_clean":"2024-09-16T13:31:31.799202+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:31:31.799202+0000","last_undegraded":"2024-09-16T13:31:31.799202+0000","last_fullsized":"2024-09-16T13:31:31.799202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'528","last_scrub_stamp":"2024-09-16T13:31:31.799089+0000","last_deep_scrub":"166'528","last_deep_scrub_stamp":"2024-09-16T13:31:31.799089+0000","last_clean_scrub_stamp":"2024-09-16T13:31:31.799089+0000","objects_scrubbed":16,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:53.490903+0000","scrub_duration":0.012917092999999999,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"166'469","reported_seq":3192,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.456588+0000","last_change":"2024-09-16T13:31:15.456540+0000","last_active":"2024-09-16T13:31:15.456588+0000","last_peered":"2024-09-16T13:31:15.456588+0000","last_clean":"2024-09-16T13:31:15.456588+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:31:15.456588+0000","last_undegraded":"2024-09-16T13:31:15.456588+0000","last_fullsized":"2024-09-16T13:31:15.456588+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'469","last_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_deep_scrub":"166'469","last_deep_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.456434+0000","objects_scrubbed":12,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:21:06.510924+0000","scrub_duration":0.010685587999999999,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"166'382","reported_seq":1833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.868714+0000","last_change":"2024-09-16T13:31:12.868676+0000","last_active":"2024-09-16T13:31:12.868714+0000","last_peered":"2024-09-16T13:31:12.868714+0000","last_clean":"2024-09-16T13:31:12.868714+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:31:12.868714+0000","last_undegraded":"2024-09-16T13:31:12.868714+0000","last_fullsized":"2024-09-16T13:31:12.868714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'382","last_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_deep_scrub":"166'382","last_deep_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.868550+0000","objects_scrubbed":17,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:10.566793+0000","scrub_duration":0.012531097999999999,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"166'859","reported_seq":3466,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.652501+0000","last_change":"2024-09-16T13:31:15.652472+0000","last_active":"2024-09-16T13:31:15.652501+0000","last_peered":"2024-09-16T13:31:15.652501+0000","last_clean":"2024-09-16T13:31:15.652501+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:31:15.652501+0000","last_undegraded":"2024-09-16T13:31:15.652501+0000","last_fullsized":"2024-09-16T13:31:15.652501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'859","last_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_deep_scrub":"166'859","last_deep_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.652383+0000","objects_scrubbed":11,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:42.495261+0000","scrub_duration":0.010275467,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18740,"ondisk_log_size":18740,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":336,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":336,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509488,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303496,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068344,"statfs":{"total":96636764160,"available":96325984256,"internally_reserved":0,"allocated":2031616,"data_stored":1606469,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46200000000000002,"15min":0.47599999999999998},"min":{"1min":0.32300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.503}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46400000000000002,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.192,"15min":0.192},"max":{"1min":0.64800000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.61499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67700000000000005,"5min":0.879,"15min":0.879},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59699999999999998,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67600000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.66800000000000004}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38600000000000001,"15min":0.40400000000000003},"min":{"1min":0.23999999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.307}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60499999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65200000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.42599999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509490,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303468,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068372,"statfs":{"total":96636764160,"available":96326012928,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.34699999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0029999999999999,"5min":2.169,"15min":2.169},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.45300000000000001,"15min":0.45200000000000001},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.39500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.68400000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.54000000000000004},"min":{"1min":0.33700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.93600000000000005,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.496,"15min":0.496},"min":{"1min":0.371,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40400000000000003,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.64800000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64300000000000002,"5min":0.752,"15min":0.752},"last":0.378}]},{"osd":6,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44400000000000001,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.495},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.47599999999999998},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.29799999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.877,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.35999999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509489,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303468,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068372,"statfs":{"total":96636764160,"available":96326012928,"internally_reserved":0,"allocated":2002944,"data_stored":1016286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.71599999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68799999999999994,"5min":1.173,"15min":1.173},"last":0.65700000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.501,"15min":0.51700000000000002},"min":{"1min":0.36799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70099999999999996,"5min":1.224,"15min":1.224},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64600000000000002,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.38400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63600000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.752}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.49399999999999999},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66700000000000004,"5min":1.296,"15min":1.296},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.67100000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64600000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68899999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68300000000000005,"5min":0.747,"15min":0.747},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72299999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.67200000000000004}]}]},{"osd":3,"up_from":15,"seq":64424509488,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311788,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060052,"statfs":{"total":96636764160,"available":96317493248,"internally_reserved":0,"allocated":2002944,"data_stored":1016290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.38100000000000001},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35699999999999998,"15min":0.376},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.59299999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36399999999999999,"15min":0.375},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.504,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":1.093,"15min":1.093},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.38200000000000001,"15min":0.39000000000000001},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.47699999999999998,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.71599999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70999999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.622},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70399999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68999999999999995,"5min":1.119,"15min":1.119},"last":0.69499999999999995}]}]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299296,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072544,"statfs":{"total":96636764160,"available":96330285056,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.66700000000000004,"15min":0.75600000000000001},"min":{"1min":0.34799999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.67400000000000004,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.498,"15min":0.496},"min":{"1min":0.39300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.55100000000000005},"min":{"1min":0.36899999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":1.286,"15min":1.286},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71799999999999997,"5min":1.236,"15min":1.236},"last":0.50900000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.371,"5min":0.245,"15min":0.245},"max":{"1min":0.61399999999999999,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.496},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64900000000000002,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":1.288,"15min":1.288},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.433},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68999999999999995,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.35699999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47499999999999998,"15min":0.503},"min":{"1min":0.314,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70699999999999996,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72699999999999998,"5min":1.008,"15min":1.008},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47999999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542193,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307616,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064224,"statfs":{"total":96636764160,"available":96321765376,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33300000000000002,"15min":0.34300000000000003},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.50600000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.60499999999999998,"15min":0.72499999999999998},"min":{"1min":0.26100000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.47499999999999998,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34399999999999997,"15min":0.34999999999999998},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.60399999999999998,"15min":0.72199999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.47999999999999998,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38800000000000001,"15min":0.40200000000000002},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46300000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.379},{"interface":"front","average":{"1min":0.378,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.28199999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.53700000000000003,"5min":1.194,"15min":1.194},"last":0.45200000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54800000000000004,"15min":0.57099999999999995},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79000000000000004,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67100000000000004,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.751,"5min":1.321,"15min":1.321},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.66000000000000003,"5min":1.23,"15min":1.23},"last":0.442}]},{"osd":6,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.77700000000000002,"15min":0.88500000000000001},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69699999999999995,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77200000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.79100000000000004,"15min":0.90200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67600000000000005,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72799999999999998,"5min":1.347,"15min":1.347},"last":0.66600000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542195,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1994752,"data_stored":1016120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41799999999999998,"15min":0.371},"min":{"1min":0.216,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.34000000000000002},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55600000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.38}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.42799999999999999,"15min":0.38},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.58899999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.248,"5min":0.221,"15min":0.221},"max":{"1min":0.50900000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.46700000000000003,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.37,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.498}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63700000000000001,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80100000000000005,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.621},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.64100000000000001,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.38,"15min":0.38},"max":{"1min":0.871,"5min":8.298,"15min":8.298},"last":0.68700000000000006}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.40999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":1.181,"15min":1.181},"last":0.65300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542193,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299360,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072480,"statfs":{"total":96636764160,"available":96330219520,"internally_reserved":0,"allocated":2056192,"data_stored":1606635,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.442,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.35799999999999998,"15min":0.36399999999999999},"min":{"1min":0.29099999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33900000000000002},"min":{"1min":0.224,"5min":0.192,"15min":0.192},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65100000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.377,"15min":0.379},"min":{"1min":0.255,"5min":0.249,"15min":0.249},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71099999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79400000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.56100000000000005},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78600000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81200000000000006,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78000000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:31:44.518 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-16T13:28:25.959435+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=28, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=30, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:31:44.519 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:32:04.519 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:32:04.795 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T13:32:04.796 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T13:32:04.807 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-16T13:32:03.884447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590908,"num_objects":316,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":302,"num_read":166042,"num_read_kb":804180,"num_write":21785,"num_write_kb":462551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":63143333,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":502,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18842,"ondisk_log_size":18842,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2440288,"kb_used_data":15712,"kb_used_omap":13,"kb_used_meta":2424498,"kb_avail":752534432,"statfs":{"total":773094113280,"available":770595258368,"internally_reserved":0,"allocated":16089088,"data_stored":9310624,"data_compressed":12029,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":13632,"internal_metadata":2482686656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":166,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":166,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002581"},"pg_stats":[{"pgid":"4.1f","version":"166'378","reported_seq":2287,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.175418+0000","last_change":"2024-09-16T13:31:08.175392+0000","last_active":"2024-09-16T13:31:08.175418+0000","last_peered":"2024-09-16T13:31:08.175418+0000","last_clean":"2024-09-16T13:31:08.175418+0000","last_became_active":"2024-09-16T13:28:26.986763+0000","last_became_peered":"2024-09-16T13:28:26.986763+0000","last_unstale":"2024-09-16T13:31:08.175418+0000","last_undegraded":"2024-09-16T13:31:08.175418+0000","last_fullsized":"2024-09-16T13:31:08.175418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'378","last_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_deep_scrub":"166'378","last_deep_scrub_stamp":"2024-09-16T13:31:08.175307+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.175307+0000","objects_scrubbed":11,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:49.299655+0000","scrub_duration":0.010270289,"objects_trimmed":0,"snaptrim_duration":0.000388141,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2829,"num_read_kb":20114,"num_write":357,"num_write_kb":9432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"166'1105","reported_seq":11867,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.010295+0000","last_change":"2024-09-16T13:31:16.010249+0000","last_active":"2024-09-16T13:31:16.010295+0000","last_peered":"2024-09-16T13:31:16.010295+0000","last_clean":"2024-09-16T13:31:16.010295+0000","last_became_active":"2024-09-16T13:29:09.920091+0000","last_became_peered":"2024-09-16T13:29:09.920091+0000","last_unstale":"2024-09-16T13:31:16.010295+0000","last_undegraded":"2024-09-16T13:31:16.010295+0000","last_fullsized":"2024-09-16T13:31:16.010295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1105","last_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_deep_scrub":"166'1105","last_deep_scrub_stamp":"2024-09-16T13:31:16.010149+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.010149+0000","objects_scrubbed":10,"log_size":1105,"log_dups_size":0,"ondisk_log_size":1105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:32.387613+0000","scrub_duration":0.009411081,"objects_trimmed":0,"snaptrim_duration":0.00039545900000000002,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":15009,"num_read_kb":31079,"num_write":1668,"num_write_kb":9692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":3754456,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"166'458","reported_seq":1702,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:30.418020+0000","last_change":"2024-09-16T13:31:30.417988+0000","last_active":"2024-09-16T13:31:30.418020+0000","last_peered":"2024-09-16T13:31:30.418020+0000","last_clean":"2024-09-16T13:31:30.418020+0000","last_became_active":"2024-09-16T13:28:26.982517+0000","last_became_peered":"2024-09-16T13:28:26.982517+0000","last_unstale":"2024-09-16T13:31:30.418020+0000","last_undegraded":"2024-09-16T13:31:30.418020+0000","last_fullsized":"2024-09-16T13:31:30.418020+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'458","last_scrub_stamp":"2024-09-16T13:31:30.417882+0000","last_deep_scrub":"166'458","last_deep_scrub_stamp":"2024-09-16T13:31:30.417882+0000","last_clean_scrub_stamp":"2024-09-16T13:31:30.417882+0000","objects_scrubbed":10,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:12:24.945508+0000","scrub_duration":0.0087942999999999997,"objects_trimmed":0,"snaptrim_duration":0.0018029890000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":2055,"num_read_kb":8165,"num_write":449,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"166'515","reported_seq":4617,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.637148+0000","last_change":"2024-09-16T13:31:13.637117+0000","last_active":"2024-09-16T13:31:13.637148+0000","last_peered":"2024-09-16T13:31:13.637148+0000","last_clean":"2024-09-16T13:31:13.637148+0000","last_became_active":"2024-09-16T13:28:26.985244+0000","last_became_peered":"2024-09-16T13:28:26.985244+0000","last_unstale":"2024-09-16T13:31:13.637148+0000","last_undegraded":"2024-09-16T13:31:13.637148+0000","last_fullsized":"2024-09-16T13:31:13.637148+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'515","last_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_deep_scrub":"166'515","last_deep_scrub_stamp":"2024-09-16T13:31:13.637026+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.637026+0000","objects_scrubbed":5,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:05.922562+0000","scrub_duration":0.0094649629999999998,"objects_trimmed":0,"snaptrim_duration":0.00042815699999999997,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":7647,"num_read_kb":26945,"num_write":516,"num_write_kb":9874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"166'484","reported_seq":2396,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:39.864339+0000","last_change":"2024-09-16T13:31:39.864313+0000","last_active":"2024-09-16T13:31:39.864339+0000","last_peered":"2024-09-16T13:31:39.864339+0000","last_clean":"2024-09-16T13:31:39.864339+0000","last_became_active":"2024-09-16T13:28:26.985303+0000","last_became_peered":"2024-09-16T13:28:26.985303+0000","last_unstale":"2024-09-16T13:31:39.864339+0000","last_undegraded":"2024-09-16T13:31:39.864339+0000","last_fullsized":"2024-09-16T13:31:39.864339+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'484","last_scrub_stamp":"2024-09-16T13:31:39.864224+0000","last_deep_scrub":"166'484","last_deep_scrub_stamp":"2024-09-16T13:31:39.864224+0000","last_clean_scrub_stamp":"2024-09-16T13:31:39.864224+0000","objects_scrubbed":7,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:00.528357+0000","scrub_duration":0.0065189139999999998,"objects_trimmed":0,"snaptrim_duration":0.00090313000000000004,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3240,"num_read_kb":15629,"num_write":536,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"166'433","reported_seq":4357,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:26.910273+0000","last_change":"2024-09-16T13:31:26.910245+0000","last_active":"2024-09-16T13:31:26.910273+0000","last_peered":"2024-09-16T13:31:26.910273+0000","last_clean":"2024-09-16T13:31:26.910273+0000","last_became_active":"2024-09-16T13:28:26.984677+0000","last_became_peered":"2024-09-16T13:28:26.984677+0000","last_unstale":"2024-09-16T13:31:26.910273+0000","last_undegraded":"2024-09-16T13:31:26.910273+0000","last_fullsized":"2024-09-16T13:31:26.910273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'433","last_scrub_stamp":"2024-09-16T13:31:26.910152+0000","last_deep_scrub":"166'433","last_deep_scrub_stamp":"2024-09-16T13:31:26.910152+0000","last_clean_scrub_stamp":"2024-09-16T13:31:26.910152+0000","objects_scrubbed":11,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:39.628602+0000","scrub_duration":0.0084123670000000005,"objects_trimmed":0,"snaptrim_duration":0.00054304899999999998,"stat_sum":{"num_bytes":19,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":10,"num_read":4127,"num_read_kb":16570,"num_write":460,"num_write_kb":6968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"166'417","reported_seq":3537,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:24.386457+0000","last_change":"2024-09-16T13:31:24.386420+0000","last_active":"2024-09-16T13:31:24.386457+0000","last_peered":"2024-09-16T13:31:24.386457+0000","last_clean":"2024-09-16T13:31:24.386457+0000","last_became_active":"2024-09-16T13:28:26.985132+0000","last_became_peered":"2024-09-16T13:28:26.985132+0000","last_unstale":"2024-09-16T13:31:24.386457+0000","last_undegraded":"2024-09-16T13:31:24.386457+0000","last_fullsized":"2024-09-16T13:31:24.386457+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'417","last_scrub_stamp":"2024-09-16T13:31:24.386338+0000","last_deep_scrub":"166'417","last_deep_scrub_stamp":"2024-09-16T13:31:24.386338+0000","last_clean_scrub_stamp":"2024-09-16T13:31:24.386338+0000","objects_scrubbed":10,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:51.226800+0000","scrub_duration":0.0067827499999999997,"objects_trimmed":0,"snaptrim_duration":0.00058536700000000005,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":3419,"num_read_kb":15850,"num_write":397,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"166'422","reported_seq":3595,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.770334+0000","last_change":"2024-09-16T13:31:10.770307+0000","last_active":"2024-09-16T13:31:10.770334+0000","last_peered":"2024-09-16T13:31:10.770334+0000","last_clean":"2024-09-16T13:31:10.770334+0000","last_became_active":"2024-09-16T13:28:26.982969+0000","last_became_peered":"2024-09-16T13:28:26.982969+0000","last_unstale":"2024-09-16T13:31:10.770334+0000","last_undegraded":"2024-09-16T13:31:10.770334+0000","last_fullsized":"2024-09-16T13:31:10.770334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'422","last_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_deep_scrub":"166'422","last_deep_scrub_stamp":"2024-09-16T13:31:10.770204+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.770204+0000","objects_scrubbed":6,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:32.183077+0000","scrub_duration":0.0061710920000000004,"objects_trimmed":0,"snaptrim_duration":0.001109208,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3710,"num_read_kb":33033,"num_write":411,"num_write_kb":30750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"166'427","reported_seq":3294,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:22.399595+0000","last_change":"2024-09-16T13:31:22.399560+0000","last_active":"2024-09-16T13:31:22.399595+0000","last_peered":"2024-09-16T13:31:22.399595+0000","last_clean":"2024-09-16T13:31:22.399595+0000","last_became_active":"2024-09-16T13:29:07.182360+0000","last_became_peered":"2024-09-16T13:29:07.182360+0000","last_unstale":"2024-09-16T13:31:22.399595+0000","last_undegraded":"2024-09-16T13:31:22.399595+0000","last_fullsized":"2024-09-16T13:31:22.399595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'427","last_scrub_stamp":"2024-09-16T13:31:22.399465+0000","last_deep_scrub":"166'427","last_deep_scrub_stamp":"2024-09-16T13:31:22.399465+0000","last_clean_scrub_stamp":"2024-09-16T13:31:22.399465+0000","objects_scrubbed":15,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:37.784900+0000","scrub_duration":0.011867277000000001,"objects_trimmed":0,"snaptrim_duration":0.001864685,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4459,"num_read_kb":29774,"num_write":365,"num_write_kb":17571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":16832042,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'661","reported_seq":2902,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.003871+0000","last_change":"2024-09-16T13:31:13.003826+0000","last_active":"2024-09-16T13:31:13.003871+0000","last_peered":"2024-09-16T13:31:13.003871+0000","last_clean":"2024-09-16T13:31:13.003871+0000","last_became_active":"2024-09-16T13:28:26.984986+0000","last_became_peered":"2024-09-16T13:28:26.984986+0000","last_unstale":"2024-09-16T13:31:13.003871+0000","last_undegraded":"2024-09-16T13:31:13.003871+0000","last_fullsized":"2024-09-16T13:31:13.003871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'661","last_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_deep_scrub":"166'661","last_deep_scrub_stamp":"2024-09-16T13:31:13.003719+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.003719+0000","objects_scrubbed":7,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:28.799115+0000","scrub_duration":0.010013117,"objects_trimmed":0,"snaptrim_duration":0.00070003700000000003,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4642,"num_read_kb":15931,"num_write":791,"num_write_kb":9984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"166'843","reported_seq":4157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:22.466996+0000","last_change":"2024-09-16T13:31:22.466960+0000","last_active":"2024-09-16T13:31:22.466996+0000","last_peered":"2024-09-16T13:31:22.466996+0000","last_clean":"2024-09-16T13:31:22.466996+0000","last_became_active":"2024-09-16T13:30:51.196571+0000","last_became_peered":"2024-09-16T13:30:51.196571+0000","last_unstale":"2024-09-16T13:31:22.466996+0000","last_undegraded":"2024-09-16T13:31:22.466996+0000","last_fullsized":"2024-09-16T13:31:22.466996+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'843","last_scrub_stamp":"2024-09-16T13:31:22.466858+0000","last_deep_scrub":"166'843","last_deep_scrub_stamp":"2024-09-16T13:31:22.466858+0000","last_clean_scrub_stamp":"2024-09-16T13:31:22.466858+0000","objects_scrubbed":9,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:55.207634+0000","scrub_duration":0.0084471040000000004,"objects_trimmed":2,"snaptrim_duration":0.010124489,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5802,"num_read_kb":34366,"num_write":1223,"num_write_kb":18017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":18777278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"166'499","reported_seq":3535,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.343908+0000","last_change":"2024-09-16T13:31:14.343867+0000","last_active":"2024-09-16T13:31:14.343908+0000","last_peered":"2024-09-16T13:31:14.343908+0000","last_clean":"2024-09-16T13:31:14.343908+0000","last_became_active":"2024-09-16T13:28:26.982935+0000","last_became_peered":"2024-09-16T13:28:26.982935+0000","last_unstale":"2024-09-16T13:31:14.343908+0000","last_undegraded":"2024-09-16T13:31:14.343908+0000","last_fullsized":"2024-09-16T13:31:14.343908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'499","last_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_deep_scrub":"166'499","last_deep_scrub_stamp":"2024-09-16T13:31:14.343753+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.343753+0000","objects_scrubbed":10,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:54.979530+0000","scrub_duration":0.011985519,"objects_trimmed":0,"snaptrim_duration":0.00085238600000000001,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":4926,"num_read_kb":20482,"num_write":504,"num_write_kb":8927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'388","reported_seq":2046,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.669523+0000","last_change":"2024-09-16T13:31:04.669493+0000","last_active":"2024-09-16T13:31:04.669523+0000","last_peered":"2024-09-16T13:31:04.669523+0000","last_clean":"2024-09-16T13:31:04.669523+0000","last_became_active":"2024-09-16T13:28:26.982706+0000","last_became_peered":"2024-09-16T13:28:26.982706+0000","last_unstale":"2024-09-16T13:31:04.669523+0000","last_undegraded":"2024-09-16T13:31:04.669523+0000","last_fullsized":"2024-09-16T13:31:04.669523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'388","last_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_deep_scrub":"166'388","last_deep_scrub_stamp":"2024-09-16T13:31:04.669404+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.669404+0000","objects_scrubbed":11,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:05.964062+0000","scrub_duration":0.010041550999999999,"objects_trimmed":0,"snaptrim_duration":0.00146203,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1521,"num_read_kb":22420,"num_write":350,"num_write_kb":11070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'837","reported_seq":4384,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.260438+0000","last_change":"2024-09-16T13:31:14.260402+0000","last_active":"2024-09-16T13:31:14.260438+0000","last_peered":"2024-09-16T13:31:14.260438+0000","last_clean":"2024-09-16T13:31:14.260438+0000","last_became_active":"2024-09-16T13:28:26.984413+0000","last_became_peered":"2024-09-16T13:28:26.984413+0000","last_unstale":"2024-09-16T13:31:14.260438+0000","last_undegraded":"2024-09-16T13:31:14.260438+0000","last_fullsized":"2024-09-16T13:31:14.260438+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'837","last_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_deep_scrub":"166'837","last_deep_scrub_stamp":"2024-09-16T13:31:14.260308+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.260308+0000","objects_scrubbed":10,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:12.589220+0000","scrub_duration":0.0088603329999999998,"objects_trimmed":0,"snaptrim_duration":0.001074034,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8926,"num_read_kb":20822,"num_write":991,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"166'794","reported_seq":4888,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.408576+0000","last_change":"2024-09-16T13:31:05.408548+0000","last_active":"2024-09-16T13:31:05.408576+0000","last_peered":"2024-09-16T13:31:05.408576+0000","last_clean":"2024-09-16T13:31:05.408576+0000","last_became_active":"2024-09-16T13:29:09.919933+0000","last_became_peered":"2024-09-16T13:29:09.919933+0000","last_unstale":"2024-09-16T13:31:05.408576+0000","last_undegraded":"2024-09-16T13:31:05.408576+0000","last_fullsized":"2024-09-16T13:31:05.408576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'794","last_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_deep_scrub":"166'794","last_deep_scrub_stamp":"2024-09-16T13:31:05.408464+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.408464+0000","objects_scrubbed":9,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:04.309754+0000","scrub_duration":0.0084311639999999997,"objects_trimmed":0,"snaptrim_duration":0.0015253549999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5953,"num_read_kb":16253,"num_write":968,"num_write_kb":25104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":2622924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"166'772","reported_seq":3960,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:19.399335+0000","last_change":"2024-09-16T13:31:19.399306+0000","last_active":"2024-09-16T13:31:19.399335+0000","last_peered":"2024-09-16T13:31:19.399335+0000","last_clean":"2024-09-16T13:31:19.399335+0000","last_became_active":"2024-09-16T13:28:26.983566+0000","last_became_peered":"2024-09-16T13:28:26.983566+0000","last_unstale":"2024-09-16T13:31:19.399335+0000","last_undegraded":"2024-09-16T13:31:19.399335+0000","last_fullsized":"2024-09-16T13:31:19.399335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'772","last_scrub_stamp":"2024-09-16T13:31:19.399215+0000","last_deep_scrub":"166'772","last_deep_scrub_stamp":"2024-09-16T13:31:19.399215+0000","last_clean_scrub_stamp":"2024-09-16T13:31:19.399215+0000","objects_scrubbed":10,"log_size":772,"log_dups_size":0,"ondisk_log_size":772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:34.493382+0000","scrub_duration":0.010399172999999999,"objects_trimmed":0,"snaptrim_duration":0.00061661799999999998,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":8429,"num_read_kb":21312,"num_write":1006,"num_write_kb":9541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":185,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.136136+0000","last_change":"2024-09-16T13:31:05.136109+0000","last_active":"2024-09-16T13:31:05.136136+0000","last_peered":"2024-09-16T13:31:05.136136+0000","last_clean":"2024-09-16T13:31:05.136136+0000","last_became_active":"2024-09-16T13:29:07.181852+0000","last_became_peered":"2024-09-16T13:29:07.181852+0000","last_unstale":"2024-09-16T13:31:05.136136+0000","last_undegraded":"2024-09-16T13:31:05.136136+0000","last_fullsized":"2024-09-16T13:31:05.136136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.136016+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.136016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:13.629452+0000","scrub_duration":0.001343777,"objects_trimmed":0,"snaptrim_duration":0.00030205400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"166'510","reported_seq":3642,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:27.873288+0000","last_change":"2024-09-16T13:31:27.873250+0000","last_active":"2024-09-16T13:31:27.873288+0000","last_peered":"2024-09-16T13:31:27.873288+0000","last_clean":"2024-09-16T13:31:27.873288+0000","last_became_active":"2024-09-16T13:28:26.982293+0000","last_became_peered":"2024-09-16T13:28:26.982293+0000","last_unstale":"2024-09-16T13:31:27.873288+0000","last_undegraded":"2024-09-16T13:31:27.873288+0000","last_fullsized":"2024-09-16T13:31:27.873288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'510","last_scrub_stamp":"2024-09-16T13:31:27.873151+0000","last_deep_scrub":"166'510","last_deep_scrub_stamp":"2024-09-16T13:31:27.873151+0000","last_clean_scrub_stamp":"2024-09-16T13:31:27.873151+0000","objects_scrubbed":9,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:30:24.359148+0000","scrub_duration":0.010434084999999999,"objects_trimmed":0,"snaptrim_duration":0.0011532109999999999,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":3521,"num_read_kb":37836,"num_write":592,"num_write_kb":17616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.198348+0000","last_change":"2024-09-16T13:31:02.198329+0000","last_active":"2024-09-16T13:31:02.198348+0000","last_peered":"2024-09-16T13:31:02.198348+0000","last_clean":"2024-09-16T13:31:02.198348+0000","last_became_active":"2024-09-16T13:28:10.311026+0000","last_became_peered":"2024-09-16T13:28:10.311026+0000","last_unstale":"2024-09-16T13:31:02.198348+0000","last_undegraded":"2024-09-16T13:31:02.198348+0000","last_fullsized":"2024-09-16T13:31:02.198348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:02.198261+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.198261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:06.779026+0000","scrub_duration":0.0015109539999999999,"objects_trimmed":0,"snaptrim_duration":0.00047061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"166'371","reported_seq":2677,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.399969+0000","last_change":"2024-09-16T13:31:11.399927+0000","last_active":"2024-09-16T13:31:11.399969+0000","last_peered":"2024-09-16T13:31:11.399969+0000","last_clean":"2024-09-16T13:31:11.399969+0000","last_became_active":"2024-09-16T13:29:10.916341+0000","last_became_peered":"2024-09-16T13:29:10.916341+0000","last_unstale":"2024-09-16T13:31:11.399969+0000","last_undegraded":"2024-09-16T13:31:11.399969+0000","last_fullsized":"2024-09-16T13:31:11.399969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'371","last_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_deep_scrub":"166'371","last_deep_scrub_stamp":"2024-09-16T13:31:11.399838+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.399838+0000","objects_scrubbed":11,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:03.104907+0000","scrub_duration":0.010935642000000001,"objects_trimmed":0,"snaptrim_duration":0.001630259,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":2321,"num_read_kb":15244,"num_write":430,"num_write_kb":6756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":3216867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:14.157050+0000","last_change":"2024-09-16T13:31:14.157023+0000","last_active":"2024-09-16T13:31:14.157050+0000","last_peered":"2024-09-16T13:31:14.157050+0000","last_clean":"2024-09-16T13:31:14.157050+0000","last_became_active":"2024-09-16T13:28:28.093004+0000","last_became_peered":"2024-09-16T13:28:28.093004+0000","last_unstale":"2024-09-16T13:31:14.157050+0000","last_undegraded":"2024-09-16T13:31:14.157050+0000","last_fullsized":"2024-09-16T13:31:14.157050+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:14.156924+0000","last_clean_scrub_stamp":"2024-09-16T13:31:14.156924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:05:39.014384+0000","scrub_duration":0.001819235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"166'834","reported_seq":4271,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.992013+0000","last_change":"2024-09-16T13:31:04.991988+0000","last_active":"2024-09-16T13:31:04.992013+0000","last_peered":"2024-09-16T13:31:04.992013+0000","last_clean":"2024-09-16T13:31:04.992013+0000","last_became_active":"2024-09-16T13:28:26.981714+0000","last_became_peered":"2024-09-16T13:28:26.981714+0000","last_unstale":"2024-09-16T13:31:04.992013+0000","last_undegraded":"2024-09-16T13:31:04.992013+0000","last_fullsized":"2024-09-16T13:31:04.992013+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'834","last_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_deep_scrub":"166'834","last_deep_scrub_stamp":"2024-09-16T13:31:04.991902+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.991902+0000","objects_scrubbed":13,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:19.253050+0000","scrub_duration":0.01179998,"objects_trimmed":0,"snaptrim_duration":0.00025394399999999998,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":12,"num_read":5028,"num_read_kb":49623,"num_write":901,"num_write_kb":31576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.164363+0000","last_change":"2024-09-16T13:31:05.164330+0000","last_active":"2024-09-16T13:31:05.164363+0000","last_peered":"2024-09-16T13:31:05.164363+0000","last_clean":"2024-09-16T13:31:05.164363+0000","last_became_active":"2024-09-16T13:28:10.304248+0000","last_became_peered":"2024-09-16T13:28:10.304248+0000","last_unstale":"2024-09-16T13:31:05.164363+0000","last_undegraded":"2024-09-16T13:31:05.164363+0000","last_fullsized":"2024-09-16T13:31:05.164363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.164235+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.164235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:16.915040+0000","scrub_duration":0.0013633849999999999,"objects_trimmed":0,"snaptrim_duration":0.00046787400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"166'595","reported_seq":3510,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.218310+0000","last_change":"2024-09-16T13:31:12.218250+0000","last_active":"2024-09-16T13:31:12.218310+0000","last_peered":"2024-09-16T13:31:12.218310+0000","last_clean":"2024-09-16T13:31:12.218310+0000","last_became_active":"2024-09-16T13:28:26.982207+0000","last_became_peered":"2024-09-16T13:28:26.982207+0000","last_unstale":"2024-09-16T13:31:12.218310+0000","last_undegraded":"2024-09-16T13:31:12.218310+0000","last_fullsized":"2024-09-16T13:31:12.218310+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'595","last_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_deep_scrub":"166'595","last_deep_scrub_stamp":"2024-09-16T13:31:12.218143+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.218143+0000","objects_scrubbed":2,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:42:49.903676+0000","scrub_duration":0.00655574,"objects_trimmed":0,"snaptrim_duration":0.00066348100000000005,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":6330,"num_read_kb":18345,"num_write":657,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:03.612635+0000","last_change":"2024-09-16T13:31:03.612609+0000","last_active":"2024-09-16T13:31:03.612635+0000","last_peered":"2024-09-16T13:31:03.612635+0000","last_clean":"2024-09-16T13:31:03.612635+0000","last_became_active":"2024-09-16T13:28:17.926325+0000","last_became_peered":"2024-09-16T13:28:17.926325+0000","last_unstale":"2024-09-16T13:31:03.612635+0000","last_undegraded":"2024-09-16T13:31:03.612635+0000","last_fullsized":"2024-09-16T13:31:03.612635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:03.612520+0000","last_clean_scrub_stamp":"2024-09-16T13:31:03.612520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:47.994981+0000","scrub_duration":0.0014637949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"166'3","reported_seq":125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.436058+0000","last_change":"2024-09-16T13:31:05.436033+0000","last_active":"2024-09-16T13:31:05.436058+0000","last_peered":"2024-09-16T13:31:05.436058+0000","last_clean":"2024-09-16T13:31:05.436058+0000","last_became_active":"2024-09-16T13:30:51.195848+0000","last_became_peered":"2024-09-16T13:30:51.195848+0000","last_unstale":"2024-09-16T13:31:05.436058+0000","last_undegraded":"2024-09-16T13:31:05.436058+0000","last_fullsized":"2024-09-16T13:31:05.436058+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'3","last_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_deep_scrub":"166'3","last_deep_scrub_stamp":"2024-09-16T13:31:05.435938+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.435938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:04.127609+0000","scrub_duration":0.0034749310000000001,"objects_trimmed":0,"snaptrim_duration":0.00095779999999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"166'373","reported_seq":4581,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:10.203736+0000","last_change":"2024-09-16T13:31:10.203699+0000","last_active":"2024-09-16T13:31:10.203736+0000","last_peered":"2024-09-16T13:31:10.203736+0000","last_clean":"2024-09-16T13:31:10.203736+0000","last_became_active":"2024-09-16T13:28:26.986769+0000","last_became_peered":"2024-09-16T13:28:26.986769+0000","last_unstale":"2024-09-16T13:31:10.203736+0000","last_undegraded":"2024-09-16T13:31:10.203736+0000","last_fullsized":"2024-09-16T13:31:10.203736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'373","last_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_deep_scrub":"166'373","last_deep_scrub_stamp":"2024-09-16T13:31:10.203607+0000","last_clean_scrub_stamp":"2024-09-16T13:31:10.203607+0000","objects_scrubbed":4,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:13.855585+0000","scrub_duration":0.00582097,"objects_trimmed":0,"snaptrim_duration":0.000547595,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4249,"num_read_kb":35947,"num_write":393,"num_write_kb":9724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.132861+0000","last_change":"2024-09-16T13:31:06.132832+0000","last_active":"2024-09-16T13:31:06.132861+0000","last_peered":"2024-09-16T13:31:06.132861+0000","last_clean":"2024-09-16T13:31:06.132861+0000","last_became_active":"2024-09-16T13:28:17.924223+0000","last_became_peered":"2024-09-16T13:28:17.924223+0000","last_unstale":"2024-09-16T13:31:06.132861+0000","last_undegraded":"2024-09-16T13:31:06.132861+0000","last_fullsized":"2024-09-16T13:31:06.132861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:06.132727+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.132727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:25:35.050519+0000","scrub_duration":0.001337249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.558177+0000","last_change":"2024-09-16T13:31:05.558149+0000","last_active":"2024-09-16T13:31:05.558177+0000","last_peered":"2024-09-16T13:31:05.558177+0000","last_clean":"2024-09-16T13:31:05.558177+0000","last_became_active":"2024-09-16T13:28:10.377330+0000","last_became_peered":"2024-09-16T13:28:10.377330+0000","last_unstale":"2024-09-16T13:31:05.558177+0000","last_undegraded":"2024-09-16T13:31:05.558177+0000","last_fullsized":"2024-09-16T13:31:05.558177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.558059+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.558059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:07.215656+0000","scrub_duration":0.001425965,"objects_trimmed":0,"snaptrim_duration":0.00032525499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"166'492","reported_seq":3488,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.228159+0000","last_change":"2024-09-16T13:31:13.228130+0000","last_active":"2024-09-16T13:31:13.228159+0000","last_peered":"2024-09-16T13:31:13.228159+0000","last_clean":"2024-09-16T13:31:13.228159+0000","last_became_active":"2024-09-16T13:28:26.981567+0000","last_became_peered":"2024-09-16T13:28:26.981567+0000","last_unstale":"2024-09-16T13:31:13.228159+0000","last_undegraded":"2024-09-16T13:31:13.228159+0000","last_fullsized":"2024-09-16T13:31:13.228159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'492","last_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_deep_scrub":"166'492","last_deep_scrub_stamp":"2024-09-16T13:31:13.228040+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.228040+0000","objects_scrubbed":7,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:28.666805+0000","scrub_duration":0.0096771349999999999,"objects_trimmed":0,"snaptrim_duration":0.00051064899999999996,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4585,"num_read_kb":20423,"num_write":509,"num_write_kb":8581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"166'7","reported_seq":166,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:02.992584+0000","last_change":"2024-09-16T13:31:02.992512+0000","last_active":"2024-09-16T13:31:02.992584+0000","last_peered":"2024-09-16T13:31:02.992584+0000","last_clean":"2024-09-16T13:31:02.992584+0000","last_became_active":"2024-09-16T13:28:17.927331+0000","last_became_peered":"2024-09-16T13:28:17.927331+0000","last_unstale":"2024-09-16T13:31:02.992584+0000","last_undegraded":"2024-09-16T13:31:02.992584+0000","last_fullsized":"2024-09-16T13:31:02.992584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'6","last_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_deep_scrub":"166'6","last_deep_scrub_stamp":"2024-09-16T13:31:02.992436+0000","last_clean_scrub_stamp":"2024-09-16T13:31:02.992436+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:49.262037+0000","scrub_duration":0.0040630290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":249,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:05.419573+0000","last_change":"2024-09-16T13:31:05.419545+0000","last_active":"2024-09-16T13:31:05.419573+0000","last_peered":"2024-09-16T13:31:05.419573+0000","last_clean":"2024-09-16T13:31:05.419573+0000","last_became_active":"2024-09-16T13:29:07.183755+0000","last_became_peered":"2024-09-16T13:29:07.183755+0000","last_unstale":"2024-09-16T13:31:05.419573+0000","last_undegraded":"2024-09-16T13:31:05.419573+0000","last_fullsized":"2024-09-16T13:31:05.419573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:31:05.419455+0000","last_clean_scrub_stamp":"2024-09-16T13:31:05.419455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:11.274470+0000","scrub_duration":0.0019415299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"166'612","reported_seq":4108,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:06.007306+0000","last_change":"2024-09-16T13:31:06.007280+0000","last_active":"2024-09-16T13:31:06.007306+0000","last_peered":"2024-09-16T13:31:06.007306+0000","last_clean":"2024-09-16T13:31:06.007306+0000","last_became_active":"2024-09-16T13:28:26.984724+0000","last_became_peered":"2024-09-16T13:28:26.984724+0000","last_unstale":"2024-09-16T13:31:06.007306+0000","last_undegraded":"2024-09-16T13:31:06.007306+0000","last_fullsized":"2024-09-16T13:31:06.007306+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'612","last_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_deep_scrub":"166'612","last_deep_scrub_stamp":"2024-09-16T13:31:06.007192+0000","last_clean_scrub_stamp":"2024-09-16T13:31:06.007192+0000","objects_scrubbed":15,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:56.616376+0000","scrub_duration":0.011595329999999999,"objects_trimmed":0,"snaptrim_duration":0.00064612600000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":5564,"num_read_kb":41067,"num_write":549,"num_write_kb":30185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"166'9","reported_seq":168,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:04.382663+0000","last_change":"2024-09-16T13:31:04.382593+0000","last_active":"2024-09-16T13:31:04.382663+0000","last_peered":"2024-09-16T13:31:04.382663+0000","last_clean":"2024-09-16T13:31:04.382663+0000","last_became_active":"2024-09-16T13:28:18.023222+0000","last_became_peered":"2024-09-16T13:28:18.023222+0000","last_unstale":"2024-09-16T13:31:04.382663+0000","last_undegraded":"2024-09-16T13:31:04.382663+0000","last_fullsized":"2024-09-16T13:31:04.382663+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8","last_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_deep_scrub":"166'8","last_deep_scrub_stamp":"2024-09-16T13:31:04.382518+0000","last_clean_scrub_stamp":"2024-09-16T13:31:04.382518+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:07:20.475579+0000","scrub_duration":0.0084622399999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":253,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"166'2","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:19.461782+0000","last_change":"2024-09-16T13:31:19.461736+0000","last_active":"2024-09-16T13:31:19.461782+0000","last_peered":"2024-09-16T13:31:19.461782+0000","last_clean":"2024-09-16T13:31:19.461782+0000","last_became_active":"2024-09-16T13:30:51.195523+0000","last_became_peered":"2024-09-16T13:30:51.195523+0000","last_unstale":"2024-09-16T13:31:19.461782+0000","last_undegraded":"2024-09-16T13:31:19.461782+0000","last_fullsized":"2024-09-16T13:31:19.461782+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'2","last_scrub_stamp":"2024-09-16T13:31:19.461633+0000","last_deep_scrub":"166'2","last_deep_scrub_stamp":"2024-09-16T13:31:19.461633+0000","last_clean_scrub_stamp":"2024-09-16T13:31:19.461633+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:49.388690+0000","scrub_duration":0.003386072,"objects_trimmed":0,"snaptrim_duration":0.00044796700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"166'74","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:08.415903+0000","last_change":"2024-09-16T13:31:08.415862+0000","last_active":"2024-09-16T13:31:08.415903+0000","last_peered":"2024-09-16T13:31:08.415903+0000","last_clean":"2024-09-16T13:31:08.415903+0000","last_became_active":"2024-09-16T13:29:07.181686+0000","last_became_peered":"2024-09-16T13:29:07.181686+0000","last_unstale":"2024-09-16T13:31:08.415903+0000","last_undegraded":"2024-09-16T13:31:08.415903+0000","last_fullsized":"2024-09-16T13:31:08.415903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'74","last_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_deep_scrub":"166'74","last_deep_scrub_stamp":"2024-09-16T13:31:08.415767+0000","last_clean_scrub_stamp":"2024-09-16T13:31:08.415767+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:17.920374+0000","scrub_duration":0.0052567780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"166'608","reported_seq":2536,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:11.165662+0000","last_change":"2024-09-16T13:31:11.165631+0000","last_active":"2024-09-16T13:31:11.165662+0000","last_peered":"2024-09-16T13:31:11.165662+0000","last_clean":"2024-09-16T13:31:11.165662+0000","last_became_active":"2024-09-16T13:28:26.988030+0000","last_became_peered":"2024-09-16T13:28:26.988030+0000","last_unstale":"2024-09-16T13:31:11.165662+0000","last_undegraded":"2024-09-16T13:31:11.165662+0000","last_fullsized":"2024-09-16T13:31:11.165662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'608","last_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_deep_scrub":"166'608","last_deep_scrub_stamp":"2024-09-16T13:31:11.165540+0000","last_clean_scrub_stamp":"2024-09-16T13:31:11.165540+0000","objects_scrubbed":8,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:04:36.130808+0000","scrub_duration":0.0070728830000000003,"objects_trimmed":0,"snaptrim_duration":0.00047947099999999998,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3487,"num_read_kb":17250,"num_write":729,"num_write_kb":7431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"166'1262","reported_seq":5761,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:16.458253+0000","last_change":"2024-09-16T13:31:16.458211+0000","last_active":"2024-09-16T13:31:16.458253+0000","last_peered":"2024-09-16T13:31:16.458253+0000","last_clean":"2024-09-16T13:31:16.458253+0000","last_became_active":"2024-09-16T13:30:51.195923+0000","last_became_peered":"2024-09-16T13:30:51.195923+0000","last_unstale":"2024-09-16T13:31:16.458253+0000","last_undegraded":"2024-09-16T13:31:16.458253+0000","last_fullsized":"2024-09-16T13:31:16.458253+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'1262","last_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_deep_scrub":"166'1262","last_deep_scrub_stamp":"2024-09-16T13:31:16.458069+0000","last_clean_scrub_stamp":"2024-09-16T13:31:16.458069+0000","objects_scrubbed":3,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:45.766858+0000","scrub_duration":0.0047390660000000001,"objects_trimmed":0,"snaptrim_duration":0.00039869200000000002,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":9582,"num_read_kb":25606,"num_write":1824,"num_write_kb":11395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":7204226,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"166'403","reported_seq":4835,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:34.792864+0000","last_change":"2024-09-16T13:31:34.792827+0000","last_active":"2024-09-16T13:31:34.792864+0000","last_peered":"2024-09-16T13:31:34.792864+0000","last_clean":"2024-09-16T13:31:34.792864+0000","last_became_active":"2024-09-16T13:28:26.981680+0000","last_became_peered":"2024-09-16T13:28:26.981680+0000","last_unstale":"2024-09-16T13:31:34.792864+0000","last_undegraded":"2024-09-16T13:31:34.792864+0000","last_fullsized":"2024-09-16T13:31:34.792864+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'403","last_scrub_stamp":"2024-09-16T13:31:34.792734+0000","last_deep_scrub":"166'403","last_deep_scrub_stamp":"2024-09-16T13:31:34.792734+0000","last_clean_scrub_stamp":"2024-09-16T13:31:34.792734+0000","objects_scrubbed":5,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:52:15.651803+0000","scrub_duration":0.0071034569999999997,"objects_trimmed":0,"snaptrim_duration":0.00060889400000000002,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":5854,"num_read_kb":30067,"num_write":442,"num_write_kb":9469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"166'582","reported_seq":4619,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:13.179053+0000","last_change":"2024-09-16T13:31:13.179007+0000","last_active":"2024-09-16T13:31:13.179053+0000","last_peered":"2024-09-16T13:31:13.179053+0000","last_clean":"2024-09-16T13:31:13.179053+0000","last_became_active":"2024-09-16T13:28:26.985298+0000","last_became_peered":"2024-09-16T13:28:26.985298+0000","last_unstale":"2024-09-16T13:31:13.179053+0000","last_undegraded":"2024-09-16T13:31:13.179053+0000","last_fullsized":"2024-09-16T13:31:13.179053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'582","last_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_deep_scrub":"166'582","last_deep_scrub_stamp":"2024-09-16T13:31:13.178913+0000","last_clean_scrub_stamp":"2024-09-16T13:31:13.178913+0000","objects_scrubbed":10,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:45.723440+0000","scrub_duration":0.0078661749999999996,"objects_trimmed":0,"snaptrim_duration":0.00065375700000000004,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":9,"num_read":6017,"num_read_kb":43661,"num_write":587,"num_write_kb":25119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"166'434","reported_seq":5531,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:18.125960+0000","last_change":"2024-09-16T13:31:18.125928+0000","last_active":"2024-09-16T13:31:18.125960+0000","last_peered":"2024-09-16T13:31:18.125960+0000","last_clean":"2024-09-16T13:31:18.125960+0000","last_became_active":"2024-09-16T13:30:37.098313+0000","last_became_peered":"2024-09-16T13:30:37.098313+0000","last_unstale":"2024-09-16T13:31:18.125960+0000","last_undegraded":"2024-09-16T13:31:18.125960+0000","last_fullsized":"2024-09-16T13:31:18.125960+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'434","last_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_deep_scrub":"166'434","last_deep_scrub_stamp":"2024-09-16T13:31:18.125809+0000","last_clean_scrub_stamp":"2024-09-16T13:31:18.125809+0000","objects_scrubbed":9,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:49.862293+0000","scrub_duration":0.0072992939999999996,"objects_trimmed":0,"snaptrim_duration":0.00038990500000000002,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":5833,"num_read_kb":40471,"num_write":494,"num_write_kb":26185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"166'528","reported_seq":5615,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:31.799202+0000","last_change":"2024-09-16T13:31:31.799176+0000","last_active":"2024-09-16T13:31:31.799202+0000","last_peered":"2024-09-16T13:31:31.799202+0000","last_clean":"2024-09-16T13:31:31.799202+0000","last_became_active":"2024-09-16T13:28:26.985955+0000","last_became_peered":"2024-09-16T13:28:26.985955+0000","last_unstale":"2024-09-16T13:31:31.799202+0000","last_undegraded":"2024-09-16T13:31:31.799202+0000","last_fullsized":"2024-09-16T13:31:31.799202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'528","last_scrub_stamp":"2024-09-16T13:31:31.799089+0000","last_deep_scrub":"166'528","last_deep_scrub_stamp":"2024-09-16T13:31:31.799089+0000","last_clean_scrub_stamp":"2024-09-16T13:31:31.799089+0000","objects_scrubbed":16,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:53.490903+0000","scrub_duration":0.012917092999999999,"objects_trimmed":0,"snaptrim_duration":0.001417436,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":6739,"num_read_kb":25459,"num_write":576,"num_write_kb":8632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"166'469","reported_seq":3192,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.456588+0000","last_change":"2024-09-16T13:31:15.456540+0000","last_active":"2024-09-16T13:31:15.456588+0000","last_peered":"2024-09-16T13:31:15.456588+0000","last_clean":"2024-09-16T13:31:15.456588+0000","last_became_active":"2024-09-16T13:30:51.193575+0000","last_became_peered":"2024-09-16T13:30:51.193575+0000","last_unstale":"2024-09-16T13:31:15.456588+0000","last_undegraded":"2024-09-16T13:31:15.456588+0000","last_fullsized":"2024-09-16T13:31:15.456588+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'469","last_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_deep_scrub":"166'469","last_deep_scrub_stamp":"2024-09-16T13:31:15.456434+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.456434+0000","objects_scrubbed":12,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:21:06.510924+0000","scrub_duration":0.010685587999999999,"objects_trimmed":0,"snaptrim_duration":0.00039424099999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":2952,"num_read_kb":22132,"num_write":464,"num_write_kb":9816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7463525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"166'382","reported_seq":1833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:12.868714+0000","last_change":"2024-09-16T13:31:12.868676+0000","last_active":"2024-09-16T13:31:12.868714+0000","last_peered":"2024-09-16T13:31:12.868714+0000","last_clean":"2024-09-16T13:31:12.868714+0000","last_became_active":"2024-09-16T13:28:26.983215+0000","last_became_peered":"2024-09-16T13:28:26.983215+0000","last_unstale":"2024-09-16T13:31:12.868714+0000","last_undegraded":"2024-09-16T13:31:12.868714+0000","last_fullsized":"2024-09-16T13:31:12.868714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'382","last_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_deep_scrub":"166'382","last_deep_scrub_stamp":"2024-09-16T13:31:12.868550+0000","last_clean_scrub_stamp":"2024-09-16T13:31:12.868550+0000","objects_scrubbed":17,"log_size":382,"log_dups_size":0,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:10.566793+0000","scrub_duration":0.012531097999999999,"objects_trimmed":0,"snaptrim_duration":0.00081901299999999995,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":1718,"num_read_kb":17535,"num_write":414,"num_write_kb":10394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"166'859","reported_seq":3466,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-16T13:31:15.652501+0000","last_change":"2024-09-16T13:31:15.652472+0000","last_active":"2024-09-16T13:31:15.652501+0000","last_peered":"2024-09-16T13:31:15.652501+0000","last_clean":"2024-09-16T13:31:15.652501+0000","last_became_active":"2024-09-16T13:28:58.118433+0000","last_became_peered":"2024-09-16T13:28:58.118433+0000","last_unstale":"2024-09-16T13:31:15.652501+0000","last_undegraded":"2024-09-16T13:31:15.652501+0000","last_fullsized":"2024-09-16T13:31:15.652501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'859","last_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_deep_scrub":"166'859","last_deep_scrub_stamp":"2024-09-16T13:31:15.652383+0000","last_clean_scrub_stamp":"2024-09-16T13:31:15.652383+0000","objects_scrubbed":11,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:42.495261+0000","scrub_duration":0.010275467,"objects_trimmed":0,"snaptrim_duration":0.00043860500000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5502,"num_read_kb":14714,"num_write":1123,"num_write_kb":7906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2091279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":303,"num_object_clones":0,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":299,"num_read":165976,"num_read_kb":804125,"num_write":21676,"num_write_kb":460915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":61962597,"num_keys_recovered":84,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18747,"ondisk_log_size":18747,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":502,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":502,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":18459,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303496,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068344,"statfs":{"total":96636764160,"available":96325984256,"internally_reserved":0,"allocated":2031616,"data_stored":1606469,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65500000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46200000000000002,"15min":0.47599999999999998},"min":{"1min":0.32300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68500000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.46400000000000002,"15min":0.47599999999999998},"min":{"1min":0.311,"5min":0.192,"15min":0.192},"max":{"1min":0.64800000000000002,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.51200000000000001},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.61499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67700000000000005,"5min":0.879,"15min":0.879},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59699999999999998,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67600000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38600000000000001,"15min":0.40400000000000003},"min":{"1min":0.23999999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55400000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.41899999999999998},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40100000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60499999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.437}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65200000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46000000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509494,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2007040,"data_stored":1016373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.44,"15min":0.439},"min":{"1min":0.34699999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0029999999999999,"5min":2.169,"15min":2.169},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.45300000000000001,"15min":0.45200000000000001},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53000000000000003,"15min":0.54300000000000004},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.39500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.68400000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.54000000000000004},"min":{"1min":0.33700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.93600000000000005,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.496,"15min":0.496},"min":{"1min":0.371,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40400000000000003,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.64800000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.64300000000000002,"5min":0.752,"15min":0.752},"last":0.38700000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44400000000000001,"15min":0.45900000000000002},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.495},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41499999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.47599999999999998},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.29799999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.877,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.433}]}]},{"osd":4,"up_from":15,"seq":64424509493,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303472,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068368,"statfs":{"total":96636764160,"available":96326008832,"internally_reserved":0,"allocated":2007040,"data_stored":1016369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.71599999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68799999999999994,"5min":1.173,"15min":1.173},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.501,"15min":0.51700000000000002},"min":{"1min":0.36799999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70099999999999996,"5min":1.224,"15min":1.224},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64600000000000002,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.38400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.67400000000000004,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63600000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.49399999999999999},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66700000000000004,"5min":1.296,"15min":1.296},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.67100000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26500000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64600000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38900000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68899999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67400000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41399999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47499999999999998,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68300000000000005,"5min":0.747,"15min":0.747},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72299999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55100000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311792,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060048,"statfs":{"total":96636764160,"available":96317489152,"internally_reserved":0,"allocated":2007040,"data_stored":1016373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.38100000000000001},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35699999999999998,"15min":0.376},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.59299999999999997,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36399999999999999,"15min":0.375},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.504,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":1.093,"15min":1.093},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.38200000000000001,"15min":0.39000000000000001},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.47699999999999998,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.71599999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70999999999999996,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.70399999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.57899999999999996,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68999999999999995,"5min":1.119,"15min":1.119},"last":0.441}]}]},{"osd":6,"up_from":14,"seq":60129542199,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299296,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072544,"statfs":{"total":96636764160,"available":96330285056,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.66700000000000004,"15min":0.75600000000000001},"min":{"1min":0.34799999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.67400000000000004,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.498,"15min":0.496},"min":{"1min":0.39300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71699999999999997,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.55100000000000005},"min":{"1min":0.36899999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68799999999999994,"5min":1.286,"15min":1.286},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71799999999999997,"5min":1.236,"15min":1.236},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.46700000000000003},"min":{"1min":0.371,"5min":0.245,"15min":0.245},"max":{"1min":0.61399999999999999,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64900000000000002,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59999999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":1.288,"15min":1.288},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41799999999999998,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68999999999999995,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47499999999999998,"15min":0.503},"min":{"1min":0.314,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70699999999999996,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.5},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72699999999999998,"5min":1.008,"15min":1.008},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66400000000000003,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.625},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.5}]}]},{"osd":2,"up_from":14,"seq":60129542197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307616,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064224,"statfs":{"total":96636764160,"available":96321765376,"internally_reserved":0,"allocated":1990656,"data_stored":1016101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33300000000000002,"15min":0.34300000000000003},"min":{"1min":0.22800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.50600000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.60499999999999998,"15min":0.72499999999999998},"min":{"1min":0.26100000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.47499999999999998,"5min":9.3390000000000004,"15min":9.3390000000000004},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.34399999999999997,"15min":0.34999999999999998},"min":{"1min":0.23699999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.60399999999999998,"15min":0.72199999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.47999999999999998,"5min":9.4480000000000004,"15min":9.4480000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38800000000000001,"15min":0.40200000000000002},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46300000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.28199999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.53700000000000003,"5min":1.194,"15min":1.194},"last":0.40200000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54800000000000004,"15min":0.57099999999999995},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79000000000000004,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67100000000000004,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.751,"5min":1.321,"15min":1.321},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.66000000000000003,"5min":1.23,"15min":1.23},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.77700000000000002,"15min":0.88500000000000001},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69699999999999995,"5min":9.4299999999999997,"15min":9.4299999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77200000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:31:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.79100000000000004,"15min":0.90200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67600000000000005,"5min":9.5640000000000001,"15min":9.5640000000000001},"last":0.497},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72799999999999998,"5min":1.347,"15min":1.347},"last":0.59499999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311780,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060060,"statfs":{"total":96636764160,"available":96317501440,"internally_reserved":0,"allocated":1994752,"data_stored":1016120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41799999999999998,"15min":0.371},"min":{"1min":0.216,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54400000000000004,"5min":8.0999999999999996,"15min":8.0999999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.34000000000000002},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55600000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.42799999999999999,"15min":0.38},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.58899999999999997,"5min":8.2409999999999997,"15min":8.2409999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.248,"5min":0.221,"15min":0.221},"max":{"1min":0.50900000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.46700000000000003,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57399999999999995,"5min":8.2720000000000002,"15min":8.2720000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52200000000000002,"15min":0.53000000000000003},"min":{"1min":0.33000000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.37,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63700000000000001,"15min":0.58899999999999997},"min":{"1min":0.41199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80100000000000005,"5min":8.3239999999999998,"15min":8.3239999999999998},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.64100000000000001,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.38,"15min":0.38},"max":{"1min":0.871,"5min":8.298,"15min":8.298},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.40999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.628},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":1.181,"15min":1.181},"last":0.73599999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542197,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299364,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072476,"statfs":{"total":96636764160,"available":96330215424,"internally_reserved":0,"allocated":2060288,"data_stored":1606718,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32100000000000001,"15min":0.33100000000000002},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.442,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.35799999999999998,"15min":0.36399999999999999},"min":{"1min":0.29099999999999998,"5min":0.191,"15min":0.191},"max":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33900000000000002},"min":{"1min":0.224,"5min":0.192,"15min":0.192},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.27300000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.39400000000000002},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65100000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.377,"15min":0.379},"min":{"1min":0.255,"5min":0.249,"15min":0.249},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53700000000000003,"15min":0.56200000000000006},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71099999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52800000000000002,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79400000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52400000000000002,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.56100000000000005},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54500000000000004,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78600000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:31:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81200000000000006,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78000000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.73799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5599,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":6430,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:32:04.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T13:32:05.089 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:32:05.089 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:32:05.089 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T13:32:05.089 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.132 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T13:32:05.133 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T13:32:05.133 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.179 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T13:32:05.179 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T13:32:05.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.215 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T13:32:05.215 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T13:32:05.215 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.294 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T13:32:05.295 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T13:32:05.295 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.332 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T13:32:05.333 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T13:32:05.333 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.405 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T13:32:05.405 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T13:32:05.405 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.486 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T13:32:05.486 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T13:32:05.486 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.560 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T13:32:05.561 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:32:05.561 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T13:32:05.561 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.585 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T13:32:05.585 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T13:32:05.585 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.627 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T13:32:05.628 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:32:05.628 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T13:32:05.628 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.639 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T13:32:05.639 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T13:32:05.639 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.690 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T13:32:05.690 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T13:32:05.690 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:32:05.702 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T13:32:05.702 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:32:05.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-16T13:32:05.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:32:05.761 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T13:32:05.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:32:05.948 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T13:32:06.035 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:32:06.035 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T13:32:06.076 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:32:06.076 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T13:32:06.170 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:32:06.170 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T13:32:06.287 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:32:06.287 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T13:32:06.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:32:06.383 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T13:32:06.441 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:32:06.441 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T13:32:06.525 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:32:06.527 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/data/mon.a.tgz 2024-09-16T13:32:06.527 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T13:32:06.542 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.dudFqLkzp4 2024-09-16T13:32:06.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dudFqLkzp4 2024-09-16T13:32:06.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.dudFqLkzp4 2024-09-16T13:32:06.779 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.dudFqLkzp4 is 379KB 2024-09-16T13:32:06.847 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.dudFqLkzp4 2024-09-16T13:32:06.862 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/data/mon.c.tgz 2024-09-16T13:32:06.862 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T13:32:06.916 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.6kua4bMfaV 2024-09-16T13:32:06.917 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6kua4bMfaV 2024-09-16T13:32:07.092 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.6kua4bMfaV 2024-09-16T13:32:07.176 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.6kua4bMfaV is 521KB 2024-09-16T13:32:07.297 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.6kua4bMfaV 2024-09-16T13:32:07.312 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/data/mon.b.tgz 2024-09-16T13:32:07.313 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-09-16T13:32:07.331 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.H9GMiTfc4V 2024-09-16T13:32:07.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.H9GMiTfc4V 2024-09-16T13:32:07.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.H9GMiTfc4V 2024-09-16T13:32:07.610 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.H9GMiTfc4V is 521KB 2024-09-16T13:32:07.711 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.H9GMiTfc4V 2024-09-16T13:32:07.726 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:32:07.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:32:07.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:32:07.790 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T13:32:07.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:32:07.794 DEBUG:teuthology.orchestra.run.smithi163:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:32:07.819 INFO:teuthology.orchestra.run.smithi076.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:32:07.858 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:32:07.862 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T13:32:07.864 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi076/crash 2024-09-16T13:32:07.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:32:07.890 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi163/crash 2024-09-16T13:32:07.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:32:07.926 INFO:tasks.ceph:Compressing logs... 2024-09-16T13:32:07.927 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:32:07.935 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:32:07.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T13:32:07.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58069.log 2024-09-16T13:32:07.957 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46456.log/var/log/ceph/ceph-osd.0.log: 2024-09-16T13:32:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33070.log 2024-09-16T13:32:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32990.log 2024-09-16T13:32:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33070.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33070.log.gz 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.32990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33030.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32990.log.gz 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33030.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33030.log.gz 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33110.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32945.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33110.log.gz 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.32945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32945.log.gz 2024-09-16T13:32:08.010 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33150.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33150.log: 88.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33150.log.gz 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33230.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33230.log.gz 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33190.log 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr: 83.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.33190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33190.log.gz 2024-09-16T13:32:08.011 INFO:teuthology.orchestra.run.smithi163.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37487.log 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58069.log.gz 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38883.log 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46456.log.gz 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37487.log.gz 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57032.log 2024-09-16T13:32:08.085 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38883.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51290.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57032.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38715.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57859.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51290.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/ceph-client.admin.38715.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47415.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38715.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57859.log.gz 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49490.log 2024-09-16T13:32:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47415.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58319.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38419.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49490.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58319.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38419.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44818.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz --verbose 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: -- /var/log/ceph/ceph-client.admin.57607.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42109.log 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2024-09-16T13:32:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57607.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38757.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42109.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40053.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57985.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38757.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46137.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40053.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55465.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57985.log.gz 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46914.log 2024-09-16T13:32:08.088 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46137.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46676.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55465.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46914.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53501.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46676.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51493.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51494.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53501.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41489.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51493.log.gz 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56613.log 2024-09-16T13:32:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51494.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41489.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56613.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56613.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47875.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39914.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.37231.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52895.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47875.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37231.log.gz 2024-09-16T13:32:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50548.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52895.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38081.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57901.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50548.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53457.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38081.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46533.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57901.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52133.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53457.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52437.log 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46533.log.gz 2024-09-16T13:32:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56601.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52133.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37792.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52437.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56601.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37315.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37482.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37792.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37315.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52915.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37482.log.gz 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55843.log 2024-09-16T13:32:08.092 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49830.log 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52915.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37486.log 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55843.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49830.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57523.log 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 0.0% 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37486.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/ceph-client.admin.57523.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56276.log 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57523.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37273.log 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56276.log.gz 2024-09-16T13:32:08.093 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37811.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37273.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58195.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55927.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37811.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58195.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53414.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56011.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55927.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53414.log.gz 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37494.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57271.log 2024-09-16T13:32:08.094 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56011.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37494.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57355.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56274.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57271.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57355.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50328.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44479.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56274.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50328.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53036.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46177.log 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44479.log.gz 2024-09-16T13:32:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54869.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53036.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54911.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46177.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33370.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54869.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54911.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.33370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58279.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33370.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55717.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.38180.log -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42231.log 2024-09-16T13:32:08.096 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58279.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38377.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38180.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55423.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42231.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53966.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38377.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55423.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53966.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.45721.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53966.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56809.log 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-09-16T13:32:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45721.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51131.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54247.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56809.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51131.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36941.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54247.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37836.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.36941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36941.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47054.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43106.log 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37836.log.gz 2024-09-16T13:32:08.098 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47054.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53218.log 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43106.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56611.log 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53218.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50466.log 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56611.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50466.log.gz 2024-09-16T13:32:08.099 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33568.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42968.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.33568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 0.0% 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33568.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42968.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40372.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38203.log 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-09-16T13:32:08.100 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54968.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58027.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38203.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53822.log 0.0% 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54968.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57691.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58027.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53822.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56989.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55331.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57691.log.gz 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56989.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56989.log.gz -5 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-09-16T13:32:08.101 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51679.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55331.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45001.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz -5 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49310.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45382.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51679.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52833.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49310.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40154.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45382.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52833.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52833.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48672.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48595.log 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40154.log.gz 2024-09-16T13:32:08.102 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48672.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48672.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48595.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56609.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39082.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56609.log: gzip -5 --verbose -- 27.2% /var/log/ceph/ceph-client.admin.38060.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56609.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54827.log 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.0.39082.log: /var/log/ceph/ceph-client.admin.38060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38060.log.gz 2024-09-16T13:32:08.103 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54827.log: 95.0% -- replaced with /var/log/ceph/ceph-client.0.39082.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54827.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38179.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43694.log 0.0% 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57775.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38179.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50750.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43694.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57775.log.gz 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57649.log 2024-09-16T13:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38181.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50750.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57649.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57439.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38181.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43532.log/var/log/ceph/ceph-client.admin.52514.log: 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57439.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56849.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43532.log.gz 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2024-09-16T13:32:08.105 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56849.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49607.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53881.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47236.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49607.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53881.log.gz 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50650.log 2024-09-16T13:32:08.106 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47236.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54028.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50650.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49391.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42789.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54028.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57817.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49391.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56275.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42789.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57817.log.gz 2024-09-16T13:32:08.107 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56275.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53339.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38061.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44959.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53339.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50150.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38061.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49789.log 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44959.log.gz 2024-09-16T13:32:08.108 INFO:teuthology.orchestra.run.smithi076.stderr:gzip/var/log/ceph/ceph-client.admin.50150.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50150.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49789.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50247.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56990.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50247.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50247.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53624.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56990.log.gz 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58111.log 2024-09-16T13:32:08.109 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53624.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53624.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58111.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45798.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45798.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45798.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38547.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45284.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38547.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55591.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45284.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45284.log.gz 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58153.log 2024-09-16T13:32:08.110 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58153.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54008.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38461.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54008.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38461.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56473.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51817.log 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56473.log.gz 2024-09-16T13:32:08.111 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58237.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51817.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51817.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57565.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.58237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58237.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42928.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57565.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38589.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57481.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42032.log 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38589.log.gz 2024-09-16T13:32:08.112 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57481.log.gz 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51208.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42032.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44911.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56269.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51208.log.gz 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44911.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44911.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.56271.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56269.log.gz 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57187.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56271.log.gz 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57187.log.gz 2024-09-16T13:32:08.113 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41275.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.41275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41275.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr: 90.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57145.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50693.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57145.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56095.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.50693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50693.log.gz 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55379.log 2024-09-16T13:32:08.114 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54546.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56095.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55379.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54546.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55507.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.54744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-09-16T13:32:08.115 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.42531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43915.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56053.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43915.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43167.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56053.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53582.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43167.log.gz 2024-09-16T13:32:08.116 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.53582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53582.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48256.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55969.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57943.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48256.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55969.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57943.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40752.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56277.log 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40752.log.gz 2024-09-16T13:32:08.117 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49053.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56277.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56909.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47934.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49053.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56909.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57397.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47934.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57397.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47833.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-09-16T13:32:08.118 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.47833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47833.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51392.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.40413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51392.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46873.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48834.log 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.46873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46873.log.gz 2024-09-16T13:32:08.119 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37814.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43768.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37814.log: /var/log/ceph/ceph-client.admin.48834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48834.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37814.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37493.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56608.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43768.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37493.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55549.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55549.log: /var/log/ceph/ceph-client.admin.56608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56608.log.gz 2024-09-16T13:32:08.120 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55549.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45518.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44179.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.45518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45518.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44402.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48099.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44402.log: /var/log/ceph/ceph-client.admin.44179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44179.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44402.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43976.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48099.log: -- /var/log/ceph/ceph-client.admin.38967.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48099.log.gz 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49011.log 2024-09-16T13:32:08.121 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0%/var/log/ceph/ceph-client.admin.38967.log: -- replaced with /var/log/ceph/ceph-client.admin.43976.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.49011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49011.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27220.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57229.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33074.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57229.log: /var/log/ceph/ceph.tmp-client.admin.27220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57229.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27220.log.gz 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.33074.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46957.log 2024-09-16T13:32:08.122 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33074.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56272.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55675.log: /var/log/ceph/ceph-client.admin.46957.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46957.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56869.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56272.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56991.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37155.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56869.log: /var/log/ceph/ceph-client.admin.56991.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56869.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56991.log.gz 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43351.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37790.log 2024-09-16T13:32:08.123 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37155.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43351.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56597.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56597.log: /var/log/ceph/ceph-client.admin.37790.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56597.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37790.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38242.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37495.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38242.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47317.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37495.log.gz 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-09-16T13:32:08.124 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.41671.log: /var/log/ceph/ceph-client.admin.47317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47317.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.44037.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56273.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.38503.log: /var/log/ceph/ceph-osd.3.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52076.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56273.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52076.log.gz 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57313.log 2024-09-16T13:32:08.125 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.52975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37498.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57313.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43392.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.37498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37498.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43392.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48297.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.55250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.48297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48297.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.39636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56607.log 2024-09-16T13:32:08.126 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.43977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51617.log 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.56607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56607.log.gz 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57733.log 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51617.log.gz 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51089.log 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.57733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57733.log.gz 2024-09-16T13:32:08.127 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.51089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51089.log.gz 2024-09-16T13:32:08.128 INFO:teuthology.orchestra.run.smithi163.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T13:32:08.128 INFO:teuthology.orchestra.run.smithi076.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T13:32:08.128 INFO:teuthology.orchestra.run.smithi076.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T13:32:08.366 INFO:teuthology.orchestra.run.smithi076.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T13:32:08.496 INFO:teuthology.orchestra.run.smithi163.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T13:32:08.945 INFO:teuthology.orchestra.run.smithi076.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T13:32:09.149 INFO:teuthology.orchestra.run.smithi163.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T13:32:09.344 INFO:teuthology.orchestra.run.smithi163.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T13:32:10.105 INFO:teuthology.orchestra.run.smithi076.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T13:32:10.611 INFO:teuthology.orchestra.run.smithi076.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T13:32:10.716 INFO:teuthology.orchestra.run.smithi076.stderr: 95.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T13:32:10.718 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T13:32:10.718 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m2.770s 2024-09-16T13:32:10.718 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m8.673s 2024-09-16T13:32:10.718 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.505s 2024-09-16T13:32:10.739 INFO:teuthology.orchestra.run.smithi163.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T13:32:10.741 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T13:32:10.741 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m2.754s 2024-09-16T13:32:10.741 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m5.624s 2024-09-16T13:32:10.741 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m0.250s 2024-09-16T13:32:10.742 INFO:tasks.ceph:Archiving logs... 2024-09-16T13:32:10.742 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi076/log 2024-09-16T13:32:10.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:32:12.017 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi163/log 2024-09-16T13:32:12.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:32:12.776 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T13:32:12.785 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T13:32:12.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:32:12.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:32:12.870 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T13:32:12.879 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T13:32:12.879 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:32:12.882 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:32:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:32:12.899 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-09-16T13:32:12.902 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:32:12.903 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-09-16T13:32:12.903 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -301us[ -303us] +/- 39ms 2024-09-16T13:32:12.903 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +411us[ +410us] +/- 71ms 2024-09-16T13:32:12.903 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +108us[ +107us] +/- 62ms 2024-09-16T13:32:12.903 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 -291us[ -281us] +/- 38ms 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 +422us[ +422us] +/- 71ms 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +118us[ +118us] +/- 62ms 2024-09-16T13:32:12.904 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T13:32:12.905 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T13:32:12.914 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T13:32:12.915 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T13:32:12.925 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T13:32:13.028 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T13:32:13.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:32:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:32:13.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T13:32:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:32:13.169 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T13:32:13.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T13:32:13.189 INFO:teuthology.task.internal:Duration was 678.324732 seconds 2024-09-16T13:32:13.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T13:32:13.201 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T13:32:13.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:32:13.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:32:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:32:13.250 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T13:32:13.613 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T13:32:13.613 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T13:32:13.613 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:32:13.675 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T13:32:13.675 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:32:13.710 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T13:32:13.710 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:32:13.718 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:32:13.782 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T13:32:13.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:32:13.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:32:13.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T13:32:13.951 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T13:32:13.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:32:13.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:32:14.002 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T13:32:14.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:32:14.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:32:14.041 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-09-16T13:32:14.065 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-09-16T13:32:14.111 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:32:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:32:14.126 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:32:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:32:14.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T13:32:14.176 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T13:32:14.177 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi076 2024-09-16T13:32:14.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:32:14.218 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907305/remote/smithi163 2024-09-16T13:32:14.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:32:14.251 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T13:32:14.252 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:32:14.254 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:32:14.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T13:32:14.314 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T13:32:14.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T13:32:14.348 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T13:32:14.348 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:32:14.351 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:32:14.363 INFO:teuthology.orchestra.run.smithi076.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:32 /home/ubuntu/cephtest 2024-09-16T13:32:14.365 INFO:teuthology.orchestra.run.smithi163.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:32 /home/ubuntu/cephtest 2024-09-16T13:32:14.367 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T13:32:14.379 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T13:32:14.413 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} thrashers/cache thrashosds-health workloads/rbd_fsx_cache_writethrough} duration: 678.3247323036194 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-16T13:32:14.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T13:32:14.495 INFO:teuthology.run:pass